Collaborative Object Search and Grasp via Vision-Language Guided Reinforcement Learning

In complex horizontal environments such as shelves and cabinets, searching for and grasping the occluded target objects pose challenges for robots. This requires robots to autonomously move obstacles, discover and grasp target objects. To address this issue, we propose a hierarchical reinforcement learning method guided by vision-language model. First, the vision-language model is applied to achieve scene perception to infer the state of the target object. Then, a novel hierarchical reinforcement learning architecture is developed to complete collaborative searching and grasping. Specifically, the hierarchical strategy involves first selecting the obstacle object to be manipulated, then determining the moving direction until the target object is graspable. Meanwhile, we have designed a series of reward functions to systematically evaluate the effectiveness of the hierarchical strategy. Experiments show that the proposed method not only improves the grasping success rate but also reduces the number of robot operation steps, thereby enhancing the grasping efficiency.