Hybrid Deep Learning Method for Continuous 2-DoF Hand Motion Prediction Using Electromyography and Ultrasound

Limb loss and stroke-related motor impairments significantly impact individuals' mobility and independence, demanding advanced prosthetic and rehabilitation technologies. Surface electromyography has been widely used for prosthetic control, enabling gesture recognition through machine learning and deep learning models. However, sEMG-based systems often struggle with signal reliability and limited resolution for complex hand movements. Ultrasound has been explored as a complementary sensing modality, providing deeper muscle activity insights. This study proposes a novel deep learning approach integrating sEMG and US signals to enhance continuous 2-DoF hand motion estimation. We employ a sensor fusion technique utilizing an 8-channel sEMG armband and a handheld US probe, alongside an inertial measurement unit for ground truth data collection. Our method employs a hybrid convolutional neural network and long-short-term memory model to capture both spatial and temporal features of muscle activity, improving accuracy in predicting dynamic hand gestures. Experimental results suggest that the fusion of sEMG and US enhances continuous estimation accuracy particularly in fast and complex hand movements, compared to single-modality approach. The proposed method shows a potential of integrating hybrid sensing and deep learning for more intuitive and responsive prosthetic control systems.