XHGA: Expanding the Capabilities of Cross-Modal Wrist-Worn Devices for Multi-Task Hand Gesture Applications

Hand gesture applications (HGA) are essential for human-machine interaction. Although the existing solutions achieve good performance in specific tasks, they still face challenges when users navigate through different application contexts, i.e., requiring multi-task ability to support newly arrived HGA tasks. In this paper, we propose a novel wrist-worn multi-task HGA system named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">XHGA</i>, which can implement modal-domain combination, data-domain adaptation and label-domain extension to ensure the performance in multi-task scenarios. The system introduces a novel two-stage training strategy, i.e., task-agnostic stage to align cross-modal features from unlabeled arbitrary gestures through contrastive learning, and task-related stage to learn modality contributions with limited labeled data in specific tasks through self-attention mechanism, while achieves multi-objective recognition simultaneously by employing an adaptive loss function weighting method. Extensive experiments demonstrate that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">XHGA</i> can achieve an average accuracy of 92.7% with only using 15 labeled data per gesture under three HGA tasks. Compared with the state-of-the-art multi-modal approach, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">XHGA</i> reduces 82.7% training time, and 47.7% storage, with about 5% improvements in accuracy. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/htang0/XHGA</uri>.