Discrete Gesture Recognition Using Multimodal PPG, IMU, and Single-Channel EMG Recorded at the Wrist

Discrete hand-gesture recognition using sensors built into wrist-wearable devices could enable always-available input across a wide range of ubiquitous environments. For example, a user could flick their wrist to dismiss a phone call or tap their thumb and index fingers together to make a selection in mixed reality. To move toward such applications, this work evaluates a new multimodal commercially available device (the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BioPoint</i> by <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SIFI Labs</i>) for recognizing seven dynamic hand gestures. Three sensors were evaluated, including a single channel of electromyography (EMG), a three-axis accelerometer (ACC), and photoplethysmography (PPG). Using a deep LSTM-based network, the relative performance of each sensor and all possible combinations were compared for their gesture classification abilities. The results show that the combination of all sensors led to the highest classification accuracy (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;$</tex-math></inline-formula>96%), significantly outperforming the individual performance of each sensor (p <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt; $</tex-math></inline-formula> 0.05). In addition, the fusion of all sensors significantly improved performance across days (p <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt; $</tex-math></inline-formula> 0.05) and was significantly more resilient when classifying gestures elicited in unseen limb positions (p <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt; $</tex-math></inline-formula> 0.05). These results highlight the complementary benefits of fusing EMG, ACC, and PPG signals as a viable path forward for the reliable recognition of discrete event-driven gestures using wrist-based wearables.