DUAL-Net: Joint Domain-Invariant and User-Adaptive Feature Learning for Gesture Recognition

Human activity recognition has become an important component of human–computer interaction and ubiquitous computing. Among various sensing technologies, WiFi-based gesture recognition has attracted increasing attention due to its contactless nature and robustness to visual occlusion. However, environmental variations and user-specific differences often lead to significant performance degradation, particularly in cross-user scenarios. Existing methods primarily focus on learning domain-invariant representations, which may overlook user-specific characteristics that are essential for accurate recognition. To address this issue, we propose the Domain-invariant and User-Adaptive Learning Network (DUAL-Net), a dual-branch framework that jointly models domain-invariant and user-adaptive representations. Specifically, DUAL-Net incorporates a contrastive fusion learning (CFL) module with modality-specific encoders to learn complementary representations from WiFi and vision modalities. Furthermore, a spatial matrix difference (SMD)-guided cross-modal generation (CMG) module is introduced to generate user-adaptive WiFi features by incorporating structural priors derived from skeletal representations. To improve deployment efficiency, DUAL-Net adopts a two-stage learning framework, where adaptation is conducted offline to reduce online computational overhead. Experiments on the MM-Fi dataset and a self-collected dataset show that DUAL-Net achieves superior cross-user recognition performance compared with existing single-modality and multimodal methods. In addition, SMD-guided conditioning improves recognition accuracy by up to 8.79% over diffusion generation without structural guidance.