SafeGrasp: a secure multi-agent system for zero-shot language-driven grasp detection

Abstract Language-driven grasp detection aims to facilitate open-ended human-robot interaction. However, existing approaches still face three critical challenges: (1) difficulty in interpreting instructions involving complex spatial and logical reasoning, (2) insufficient handling of target unobstructed grasp affordance and collision risks in densely occluded scenes, and (3) limited generalization to novel scenarios due to heavy reliance on annotated data. Recent studies have integrated large Vision-Language Models (VLMs) for open-vocabulary parsing. Nevertheless, existing paradigms force the VLM to directly undertake fine-grained spatial reasoning, causing VLMs to often fail in distinguishing task-relevant objects from distractors and leading to attention shifts and visual hallucinations. We propose SafeGrasp, a zero-shot multi-agent collaborative framework. Our approach transitions the VLM from a decision-making authority to a semantic proposer, restricting it to coarse-grained task decomposition and scene parsing, while delegating all precise geometric verifications to visual toolsets. Specifically, our framework comprises three specialized agents: a supervisor agent for occlusion analysis, a Grasp agent for grasp detection, and an evaluator agent for collision verification. Extensive experiments on public datasets and in real-world scenarios demonstrate that SafeGrasp achieves competitive performance and consistent improvements over existing baselines, providing robust, collision-aware grasping without model fine-tuning.