A Multi-Model Fusion Framework to Enhance sEMG-Based Gesture Recognition Assisted by Visual Features

Gesture recognition plays an increasingly important role in fields such as human-computer inter-action and rehabilitation robots and therapy. However, most current gesture recognition systems are based on a single type of sensor, which limits the feature sources for gesture recognition. Therefore, this paper proposes a time-frequency and visual feature fusion multimodal recognition framework (TFVF -CNN). It combines the deep features extracted from a small amount of image data with the time-frequency fea-tures of surface electromyogram (sEMG) signals to improve the accuracy of gesture recognition based on sEMG. Additionally, this paper introduces a multi-modal dataset (MEVD) containing raw sEMG signal data, sEMG signal time-frequency data, and image data, providing a reliable data source for experiments. Finally, the experimental results show that gesture recognition accuracy with multimodal fusion features and time-frequency features based on sEMG signals is 93.88 % and 95.9 %, respectively. Furthermore, we verified the performance of gesture recognition under partial image occlusion and found that TFVF -CNN still exhibits superior performance.