Dual Stream Attention Convolutional Neural Network for Motion Recognition using EMG-Accelerometer Fusion Signals

Multimodal fusion methods are widely used to address the issue of inadequate description of human movement intention by relying solely on a single surface electromyography (sEMG) signal. However, simply fusing different modalities may cause a decrease in the classification accuracy of the pattern recognition system. In this study, we developed a dual stream attention convolutional neural network (DSACNN) to enhance the cross modal correlation between (sEMG) and accelerometer signals. To validate the efficacy of the proposed methods, we conducted a comparative analysis. We classified the multimodal signals of nine distinct upper limb motion patterns from 10 human subjects using our method and compared it with other multimodal fusion techniques, such as multimodal convolutional neural network (MCNN) and cross modal attention convolutional neural network (CMACNN). We also evaluated these methods in three scenarios: intra-subject, inter-subject, and inter-subject using multiple subjects' data for training. The results showed that the proposed method had a significant increase in classification accuracy in the inter - subject scenario when multiple subjects' data could train the model (p < 0.05). The proposed method could improve the ability to learn common features across different subjects significantly. Therefore, our method can significantly decrease the training burden when the subject changes and can demonstrate the clinical ability of the prosthesis.