Achieving natural and engaging verbal interactions is one of the main challenges faced by Social Robotics. In this context, physical embodiment may be one of the most critical factors: indeed, previous work indicates that physical robots elicit more favorable social responses than virtual agents. However, the effects of physical embodiment have been analysed only in some specific and limited scenarios, where verbal interaction was reduced to basic commands.The current work aims at investigating the effect of robots' physical embodiment in a pure conversation task, by considering some relevant aspects of social interaction, such as usability, speech interface quality, user satisfaction and engagement. To this aim, a pilot experiment where participants were required to chitchat with a robot and a smartphone app, both connected to the same conversation framework, has been carried out. Preliminary results are presented and discussed, and they offer interesting insights about the positive effects of physical embodiment on some of the analysed aspects.