Teleoperation control of soft pneumatic fingers based on visual target recognition of hand bending features

Abstract Soft robots possess unique operational advantages in unstructured environments. The teleoperation control of soft pneumatic fingers, incorporating visual target recognition based on hand bending characteristics, provides a technological foundation for scenarios such as medical assistance and hazardous environment operations. It further represents a new paradigm of contact-free teleoperation control. A multi-cavity soft finger was designed and fabricated using the silicone rubber material Smooth-Sil 950. A Yeoh model was utilized to establish the relationship between the input pneumatic pressure and the bending angle, and the bending characteristics of the soft finger were simulated by the finite element method. A spatial gesture feature recognition system based on OpenCV is designed by analyzing the HSV (H-Hue, S -Saturation, V–Value) color space. Analyze the image features of the human hand, identify 21 key points of the human hand based on the MediaPipe framework, and perform theoretical modeling to design a finger bending angle recognition algorithm. The fuzzy matching mapping mechanism between the human hand and the soft finger is realized by the different identification values of the matrix. The activation matrix enables independent control of individual fingers while maintaining compatibility for multi-finger coordinated control. Integrate the feedback information from the pressure sensor and the flexible bending sensor to construct the closed-loop control of the soft finger. Experimental results demonstrate that the multi-chamber type of soft pneumatic finger achieves precise controllability within 110° bending range. High-precision remote control of the soft robot is realized using only a monocular camera, while visual recognition of gesture features maintains good robustness even in complex backgrounds. It validates that closed-loop control of soft robots based on vision is an effective approach for achieving contactless teleoperation.