Leveraging LSTM, tactile sensors, and haptic feedback to augment prosthetic control via grasp type prediction and grasp type feedback

Performing grasping tasks with prosthetic hands is often a slow and clumsy affair, requiring heavy reliance on visual feedback, greatly limiting the use of prosthetic hands in daily life activities. Automating the grasping tasks via machine learning models has emerged as a promising solution. However, these methods diminish user control transforming the prosthetic hand into more of a tool than a natural extension of the body. Alternatively, this work presents a method to predict and provide haptic feedback on the prosthetic hand's current grasp, aiming to aid user decision-making and control without relying on visual cues. Soft tactile sensors and deep learning models recognize the prosthetic hand's grasp type, which is conveyed to the user through a unique haptic stimulation pattern. Long Short-Term Memory (LSTM) networks were employed for the prediction, trained on a diverse dataset of five everyday grasp types. Real-world tests using prosthetic and human hands demonstrate the approach's practicality, with confident predictions made in under one second, achieving average accuracies of 88.68% and 86.44%, respectively. The proposed approach prioritizes user control, providing real-time grasp feedback with the goal of fostering a stronger sense of embodiment despite potentially reduced grasping accuracy compared to previous approaches emphasizing automation.