AI Co-Pilot Object Recognition for Sensory Soft Robotic Grippers

Developing non-visual sensing and intelligent recognition technologies is crucial for enhancing the manipulation performance of robots in dim or obstructed environments. Although precise object recognition has been extensively researched for rigid manipulators, the adoption of these techniques in soft robotic systems has been limited by the high modulus or low sensitivity of existing sensors (gauge factor/Young’s modulus<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$&lt;$</tex-math> </inline-formula>10 kPa<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{-1}$</tex-math> </inline-formula>). Meanwhile, these systems necessitate advanced artificial intelligence(AI) algorithms to effectively process multiple sensing data. In this study, we utilize newly developed soft sensors and AI algorithm to establish a biomimetic perceptual soft gripper system capable of sensing and generating category object information during the grasping task. A strain/pressure bimodal sensor, mimicking the exceptional softness (Young’s modulus<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$&lt;$</tex-math> </inline-formula>10 kPa) and high sensitivity (gauge factor<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$&gt;$</tex-math> </inline-formula>2000) of human skin, has been developed and seamlessly integrated into a three-finger soft robotic gripper. A Swin Transformer network was developed to learn rules from bimodal data acquired from sensors and generate category information of the grasping objects. The perceptual gripper system exhibited superior recognition accuracy compared to previously reported systems, achieving an impressive 94.9% accuracy in categorizing 18 objects of varying shapes and sizes. We believe this advancement unlocks soft machines’ potential for automated applications. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i>—Non-visual sensing and recognition techniques for soft robotic grippers are valuable for operation in dimly lit or obstructed environments. Traditional rigid sensor have limitations when applied to soft robotic systems due to their high modulus. This research developed novel ultra-soft and high sensitive strain/pressure bimodal sensors integrated into a three-fingered soft robotic gripper. Meanwhile, an AI algorithm applicable to the soft gripper system was developed to process the sensing data and generate the corresponding object category information when grasping the object. With the aid of the developed soft sensor and AI algorithm, the perceptual soft gripper can achieve high-precision object recognition, giving it great potential for automating tasks such as item picking, assembly, and handling in energy-efficient warehouses and logistics centers. The developed soft sensor and AI algorithm co-pilot object recognition strategy can contribute to the advancement of soft machine automation capability technology.