Multimodal Shared Control of a Fully Wearable Prosthetic Hand/Wrist System

Expanding the input bandwidth of the human-machine interface to capture more control intentions from the human is key to achieving dexterous control of multi-degree-of-freedom prosthetic hands/wrists. This paper presents a wearable intelligent prosthesis system based on multimodal fusion, which integrates voice interaction, myoelectric control, limb movement decoding, and computer vision-based environmental perception. The system supports intention estimation throughout the entire process from grasping to operation, enabling synchronized control of 4 grasp gestures and 2 wrist DOFs. Moreover, all these decisions are made automatically during the user’s natural and continuous body movements. Six able-bodied subjects and two amputee subjects participated in a comparative experiment involving multi-object grasping and operation in a cluttered environment. Compared to myoelectric pattern recognition, our method demonstrated significant advantages in improving grasp and operation efficiency (reducing total time, grasp time, and operation time by 67.62%, 67.89%, and 67.40% for able-bodied subjects, and by 73.93%, 69.82%, and 76.67% for amputees). It also reduced control burden, with gesture and wrist myocontrol times decreasing by 71.77% and 100% in able-bodied subjects, and by 80.59% and 100% in amputees. These advantages are not limited to the grasp object, grasp part, operation type, or the subject involved. Compared to other semi-autonomous control methods, our method achieved a higher reduction in time metrics, with reduction rates ranging from 22.61% to 84.82% higher, resulting in a more significant performance improvement. Furthermore, the questionnaire showed that the system was well-recognized by the subjects in terms of operational robustness, wearing comfort, and user experience.