A Camera-Embedded Self-Adaptable Finger With Multi-Modal Sensing Capabilities for Robotic Manipulation

This work introduces a novel vision-based sensing approach for self-adaptable soft fingers using an embedded camera, demonstrating proprioceptive and exteroceptive capabilities. The system achieves normal force estimation, Z-displacement estimation, slip detection, and force application position classification by integrating the Pi Camera V3 sensor into the structure of an additively manufactured self-adaptable soft finger. Our approach avoids the deformation disambiguation required in elastomer-based sensors by enabling direct deformation sensing without additional sensing layers like a silicon pad that alters the finger’s compliant characteristics. The system also overcomes the occlusion limitations inherent in external camera-based methods by observing structural changes through strategically placed dot patterns within the finger’s internal geometry, which are visible to the embedded camera. The camera mounting position was verified using Finite Element Analysis (FEA) while ensuring finger compliance and as much deformation as possible without touching the camera. We combined two algorithms: an optical flow-based method for reliably detecting slip events and a Convolutional Neural Network (CNN) for force, displacement estimation, and position classification. The slip detection algorithm uses a dual-threshold approach that combines the Median Absolute Deviation (MAD) and standard deviation. Customized rigs integrated with a Universal Testing Machine (UTM) were used for sensor characterization, dataset acquisition, and slip detection parameter optimization. Furthermore, the system was validated through integration with an industrial gripper in an experimental sequence on different object types to mimic real-world manipulation, where both algorithms were integrated and deployed on Raspberry Pi 5 to achieve proprioceptive and exteroceptive sensing. Reliable performance was achieved with slip detection operating at 28.5 Hz and CNN update rate of 8 Hz. The experimental results demonstrate the effectiveness of the proposed system for both sensing modalities. This study presents a compact, integrated solution that maintains the inherent compliant characteristics of the self-adaptable finger while providing feedback for real-world manipulation tasks.