Temporal cross-modal fusion method of tactile, 2D visual, and point cloud data for slip detection in robotic grasping

Abstract Slip detection in robotic grasping constitutes a critical technology for ensuring stable manipulation. However, current methodologies predominantly rely on single sensing modalities while neglecting temporal correlations in multi-source data, resulting in limited generalization capabilities under complex scenarios. To address this limitation, this paper proposes a multi-modal visual-tactile fusion network for slip detection, which achieves cross-modal feature complementarity and robust slip state discrimination through coordinated temporal learning of 2D visual, point cloud, and tactile data features. To validate the method’s effectiveness, we constructed a visual-tactile dataset incorporating 2D visual, point cloud, and tactile information. The proposed model achieves a slip detection precision of 96.02% on this dataset. Furthermore, comparative experiments with baseline algorithms under varying illumination conditions demonstrate that our model maintains detection precisions of 90.52% and 85.74% in overexposed and dynamic lighting environments, respectively, outperforming baseline models. Additional robustness evaluations under occlusion environment and grasping postures confirm the model’s superior performance compared to single-modality approaches. Moreover, the proposed method demonstrates excellent performance in practical grasping experiments. These results substantiate that the proposed model can accurately determine object slip states even when confronted with missing 2D visual data, incomplete point cloud information, or degraded tactile sensor data quality, thereby providing a reliable perceptual solution for dexterous robotic manipulation.