Lightweight Vision Language Model-Guided Gesture Recognition Based on Electromyography

Neuromuscular dysfunction poses a critical health challenge, affecting both patients and caregivers. Current active rehabilitation devices relying on electromyography (EMG) face significant hurdles due to high data acquisition demands, inaccurate decoders, and model degradation from sensor distribution shifts across participants. We address these limitations with a vision language model (VLM) that pseudo-labels motor-intent predictions, enhancing decoding accuracy and reliability under sensor distribution shift across participants. Our proposed pipeline integrates visual information with EMG decoding by employing a parallel processing framework: vision data is processed through the VLM to determine user intent, object and grasp type, which in turn provides pseudo-labels for the EMG decoder’s output. The pseudo-label is then utilized to infer motor-intent, enabling precise external control of assistive devices. In our experimental analysis, our proposed pipeline demonstrated superior performance compared to an EMG-only decoder, achieving 100% decoding accuracy across four upper limb gestures in four participants despite sensor distribution shift across them. Our approach underscores the potential of integrating visual understanding with neural signal analysis to enhance the reliability and effectiveness of active rehabilitation devices, ultimately improving patient outcomes by providing more accurate and consistent assistive technologies. The project page including code can be found: https://github.com/deremustapha/VLM-EMG4Gesture.