Inferring Fine Finger Motions for Prosthetic Control: An Ultrasound-Based Approach to Real-Time Estimation of Finger Kinematics

A central challenge in building robotic prostheses is the creation of a sensor-based system able to read physiological signals from the lower limb and instruct a robotic hand to perform various tasks. Existing systems typically perform discrete gestures such as pointing or grasping, by employing electromyography (EMG) or ultrasound (U.S.) technologies to analyze muscle states. While detecting finger activation has been done in the past, we are interested in detection, or inference, done in the context of fine motions that evolve over time. Examples include motions occurring when performing fine and dexterous tasks such as typing on a keyboard or playing a musical instrument. We consider this task as an important step toward higher adoption rates of robotic prostheses among arm amputees, as it has the potential to dramatically increase functionality in performing daily tasks. To this end, we present an end-to-end robotic system, which can successfully infer fine finger motions in real-time. This is achieved by modeling the hand as a robotic manipulator and using it as an intermediate representation to encode muscles' dynamics from U.S. images. We evaluated our method by collecting data from a group of subjects and demonstrating how it can be used to replay music played on the piano or text typed on a computer keyboard. To the best of our knowledge, this is the first study demonstrating these downstream tasks within an end-to-end system.