Dexterous grasping of robotic hand posture and force generation through visual–tactile fusion

Abstract Accurately grasping objects with varying levels of hardness and softness is a challenging task. Current robotic grasping methods rely solely on visual to determine the grasping posture, without considering the grasping force of different objects, which might increase the risk of slippage or damage. For the purpose of realize the improvement and supplementation of visual information from the dimension of force, several robots include tactile sensors installed at fingertips. In this paper, we propose a method that integrates visual and tactile information to accurately determine the 6D posture and force for robot grasping. The main challenges of this research include (1) grasping representation of visual and tactile data from heterogeneous sources, (2) Construction of the implicit link between grasping posture and clutching force , (3) a visual–tactile training dataset. To address these challenges, we first use a grasping quality assessment module to filter the grasping posture and a transformer-depth separation convolution module to predict the grasping force; second, we propose a diffusion model with multi-stage convolution module to generate the optimal grasping. In addition, we created a visual–tactile grasping data representation form that included the grasping quality of grasping postures, and matching tactile data for objects of 8 distinct hardness. Experiments show that our method achieves a 98% success rate and decreases grasping force redundancy by 78% compared to purely visual methods,which significantly improves the accuracy and stability of the robot grasping.