MCS-ResNet: A Generative Robot Grasping Network Based on RGB-D Fusion

RGB and Depth (RGB-D) modalities are widely used in planar grasp detection methods. However, most previous grasp detection approaches fail to fully exploit the distinct information contained in these two modalities and simply regard depth information as part of RGB information. To better integrate RGB-D information and improve the accuracy of planar grasp detection, a novel generative multimodal fusion with channel and spatial attention residual network (MCS-ResNet) is developed, which is specifically designed for 2-D planar grasping. The proposed network has two newly designed modules and a distinctive input method that expands depth information into three channels. First, we propose a novel RGB-D fusion module to fully extract information from both RGB-D modalities by disregarding secondary information and enhancing channel and spatial sensitivity. Then, we convert single-channel depth images into three-channel images to enable the network to capture richer depth features. Finally, a novel residual module is designed to make the model pay more attention to channel-wise information. Besides, to equip the model with classification capabilities, a multiobject solid waste grasp dataset with class information is created. In the comparison experiments, MCS-ResNet obtains 95.9% grasping accuracy and 97.8% classification accuracy (CA) on the solid waste grasping dataset with classification information (SWC) dataset and 98.5% grasping accuracy on the Cornell dataset, which reflects the superiority of our model. Finally, the real-world robotic experiments were conducted in a conveyor belt scenario using an edge computing device, reaching a grasping success rate of 95.3% and a classification success rate of 96.0%.