Multi-Object Grasp Planning with Vision-Language Models for Laboratory Automation

Multi-object recognition and grasping is a critical technology in laboratory automation. However, most existing methods are limited to structured environments and exhibit poor performance in complex, unstructured, and occluded settings. To overcome these challenges in chemical experiment scenarios, this work proposes a multi-object grasping planner based on Vision-Language Models (VLM-MOGP) for robust object recognition and intelligent grasping. First, an efficient multi-task detection network (EMDN) is designed and applied for multi-object recognition and localization. Then, the image and the detected objects are fed into VLM. The planner utilizes the multimodal reasoning capability of VLM to dynamically analyze occlusion relationships and generate a collision-free grasping plan. For visually similar objects of the same category, VLM-MOGP extracts visual features through prompting to achieve object disambiguation. Experimental results demonstrate that VLM-MOGP significantly outperforms existing methods in both object recognition accuracy and grasping success rate. Additionally, experiments on real mobile robots validate the proposed approach.