Generating joint sequences with pre-trained LLMs to drive dual-arm nursing robot to perform actions

Large language models (LLMs)-driven robotic systems have become a focal point. However, current methodologies predominantly rely on auxiliary components such as pre-defined motion primitives or pre-trained skills to facilitate physical environment actions, which constitutes the primary limitation for such approaches. This study presents the SeIn framework that breaks through this limitation by leveraging prompt-conditioned sequential generation capability of LLMs to directly output control actions for dual-arm nursing robot—specifically, sequences of 14-DoF target joint positions. Firstly, the LLMs were grounded in a MuJoCo physics environment, enabling the dual-arm nursing robot to generate control actions while receiving corresponding environmental observations. Subsequently, task-specific textual prompts were designed to guide the LLMs in task reasoning, observation processing, and generation of target joint position sequences. Finally, the framework was evaluated through comprehensive MuJoCo simulations, including: ablation experiments on prompt design; expressive action experiments across seven action types spanning three operational categories; performance evaluation experiments querying four distinct LLMs at 4- and 60-s execution durations; and a functional action experiment involving Reach-to-Object. Experimental results demonstrated that SeIn framework achieved 100% success rate in generating joint sequences for actions using task-specific text prompts and pre-trained LLMs (specifically, GPT-4o and DeepSeek-V3.1), without requiring auxiliary components.