A Multimodal Framework for Estimating 6-D Force During Dynamic Human-Robot Interaction Using Deep Learning

Interaction force estimation using surface electromyography (sEMG) is a popular technique for applications such as powered exoskeletons, robotics, and rehabilitation. However, since the sEMG is nonstationary, it is difficult to estimate high-dimensional interaction forces accurately during physical human-robot interaction (pHRI). This work proposes an end-to-end 6-D force estimation framework that can accurately predict forces during dynamic pHRI from cartesian and joint space. First, the sEMG, pose, and velocity of the upper limb during pHRI are synchronously acquired. Subsequently, sEMG and kinematic information are deeply fused by tensor fusion and cross-attention fusion. Finally, a spatio-temporal neural network (STNN) is utilized to extract the features of the fused information and estimate the interaction force. Six subjects interact with the robot at four stiffness: 100, 150, 200, and 250 N/m. The proposed multimodal fusion scheme achieves excellent performance in different types of STNNs and is validated in different kinematic spaces. Among them, the highest <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$R^{2}$ </tex-math></inline-formula> of 0.969 is achieved using ConvLG in cartesian space. Compared to solely employing sEMG, the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$R^{2}$ </tex-math></inline-formula> of force estimation based on multiple modalities increases by 21.4%–42.0% (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p\lt 0.05$ </tex-math></inline-formula>). It shows the effectiveness of the presented approach and contributes a new way to estimate high-dimensional force during dynamic pHRI.