Natural Language and LLMs in Human-Robot Interaction: Performance and Challenges in a Simulated Setting

Natural language provides an intuitive and accessible way for humans to communicate with robots, fostering more natural and flexible interaction across a range of tasks. This study investigates how effectively users can command a robot using natural language within a simulated environment. By employing Gemini Flash 2.0 as the underlying Large Language Model (LLM) to interpret and translate user prompts into executable plans, we explore both the strengths and limitations of this approach. The experiments evaluated user-generated prompts across multiple predefined tasks, revealing a spectrum of outcomes — from successful task completions to errors such as misinterpretations, spatial failures, and hallucinated behaviors where the robot acted on non-existent information. The results highlight how different communication strategies, combining directive and conversational phrasing, influenced task performance. This work contributes to advancing Human-Robot Interaction (HRI) design by emphasizing the potential of LLM-powered systems while addressing the challenges of ambiguity and error resilience in user-driven command structures.