Target perception and grasping positioning system based on 2D-3D fusion

Accurate positioning of the target is the key to the robotic arm grasping task. However, due to the limitation of the hardware performance of the robotic arm, how to efficiently identify and accurately locate the target has become a major challenge, and the existing GPG grasping pose detection algorithm cannot be applied to the scene point cloud. To solve this problem, this paper proposes a target perception and grasping positioning system based on 2D-3D fusion. The system uses the YOLOv5 neural network for target recognition, combines NanoSAM to generate segmentation masks, and extracts the target's precise point cloud based on the results of the two, and inputs the GPG algorithm for grasping detection. Experimental results show that the positioning accuracy of this system exceeds 95%, and it shows good generalization ability on the Cornell grasping dataset, and can achieve efficient grasping and positioning in complex environments.