Visual–Tactile Slip Detection via Weight Allocation and Cross-Self-Attention Net

Stable grasping is a crucial skill for service robots. Detecting slip events enables robots to timely adjust their grasping strategies and enhance grip stability, facilitating precise manipulations. This study introduces the Weight Allocation and Cross-Self-Attention Network (WACSAN), a model that integrates visual and tactile data for more effective slip detection. The Weight Allocation module assigns varied weights across different positions on the feature map, particularly at the edges where contact between the object and gripper occurs—areas where the most significant changes during slipping are observed. Concurrently, the Cross-Self-Attention module extracts local information from adjacent pixels and employs multihead self-attention to integrate global information. These dual-attention mechanisms effectively capture and learn features of varying sizes, improving slip detection as objects move. Experimental results on both the Slip Detection and PoseIt datasets demonstrate that the proposed model achieves an accuracy of 96.7%, outperforming the state-of-the-art by 7.7% and confirming the model’s robustness and generalizability across different types of slip phenomena. Our study underscores the potential of multimodal data fusion in enhancing robotic grasping tasks. Our code and dataset can be accessed from the following link: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lyyao-coder/slip-detection</uri>.