A Determination Method of Robot Motion from Language Instruction with Error Correction of Motion Generation by Large Language Model

We proposed a method that, when given a language instruction, determines an appropriate robot motion procedures and the used tools for that task. In addition, our method generated a motion trajectory for a robot to execute the task. The proposed method uses a large language model (LLM) to determine robot motion procedures and tools. However, LLM may determine in inappropriate procedures and tools. For example, for the motion of “put matcha powder,” LLM may actually propose that the matcha be scooped by a natsume or poured into a bowl. The other example of output is the procedure for stirring the mixture with only matcha powder in chawan. To correct this error, our method focuses on part function. An everyday object has a role assigned to each region of that object, such as “scoop” or “stir”. The tool used for scooping must have scoop function. Thus, there are certain constraints on function and motion. The proposed method uses functional constraints and LLM to modify the correct procedures and tools. Experimental results confirmed that the proposed method was able to generate robot motion trajectory to execute a task from language instruction.