MultiGes: Real-Time Multi-Target Gesture Recognition for ISAC-Driven Human–Computer Interaction

Integrated Sensing and Communications (ISAC) integrates sensing and communication functions through ubiquitous wireless signals, providing a seamless and flexible interaction experience, making it an ideal choice for intelligent Human-Computer Interaction (HCI). Among various interaction methods, gesture recognition has garnered widespread attention. However, current RF-based gesture recognition methods within ISAC are constrained by single-target sensing and insufficient robustness. In this paper, we propose MultiGes, a real-time multi-user gesture recognition system designed to support ISAC-driven scenarios. MultiGes employs dual commercial Impulse Radio Ultra-Wideband (IR-UWB) devices to create multiple wireless links, capturing dynamic motion features from multiple targets. First, a human energy map is constructed based on the reflected signal energy to determine multi-target coordinates. Then, a Differential Human Relative Velocity (DHRV) matrix is extracted to capture fine-grained motion information. Finally, we design a lightweight STNet model to extract spatiotemporal gesture features from the DHRV matrix, enabling real-time multi-target gesture recognition. We implement the MultiGes system prototype and conduct extensive experiments on ten common gestures in HCI scenarios. Experimental results demonstrate that MultiGes achieves efficient recognition for 2 to 5 users, with an average accuracy of over 90%, providing a robust, scalable, and real-time solution for multi-target gesture recognition in ISAC-driven smart environments.