Language Reasoning in Vision-Language-Action Model for Robotic Grasping

Grasping tasks are crucial for the interaction between the robot and environment, requiring the robot to make autonomous decisions based on environmental conditions and instructions. Recently, multimodal Vision Language Models (VLMs) have demonstrated significant potential for robotic grasping. However, previous work has often been confined to high-level planning due to the gap between the text output of VLMs and the numerical predictions of robot actions, as well as the challenges of transferring rich semantic information from large models to the prediction of grasping actions. In this paper, we co-fine-tune a pre-trained VLM using the mix of large-scale visual language data and robotic grasping trajectory data, integrating model into an end-to-end robot control system to construct a Vision Language Action Model (VLA) capable of generating the robot's underlying actions. Simultaneously, to fully leverage the prior knowledge and reasoning capabilities of the pre-trained model, we insert a language reasoning structure to bridge the mapping from high-level tasks to low-level actions. This method first deduces the relevant information regarding the target object and the end-effector motions, followed by predicting the grasping actions value with finer granularity based on the language reasoning, thereby enhancing the accuracy of action predictions. A series of grasping experiments demonstrate that our method achieves a high grasping success rate under flexible language instructions and exhibits strong generalization capabilities for unseen objects and scenes.