Dynamic multi-modal attention network for robust and real-time through-wall human activity recognition

Through-wall human activity recognition (TW-HAR) has emerged as a critical area of research due to its applications in healthcare, surveillance, and emergency response. Conventional approaches relying on single-modality data, such as radar or WiFi, often face challenges in complex environments, including noise, variability in sensor placement, and environmental obstructions. These limitations are further exacerbated by factors such as signal attenuation and scattering caused by diverse wall materials (e.g., concrete, brick, drywall), misalignment between sensors and human subjects, and dynamic noise conditions, all of which significantly degrade recognition performance. This paper presents a novel Dynamic Multi-Modal Attention Network (DMAN) that integrates data from Radar, WiFi, and Acoustic sensors to achieve robust and accurate human activity recognition. The proposed framework employs a hybrid Convolutional Neural Network (CNN) and Bidirectional Long Short-Term Memory (BiLSTM) architecture, which effectively captures spatial and temporal features from multi-modal data. A dynamic attention mechanism is incorporated to prioritize critical modality-specific features, mitigating the effects of noise and redundancy. Comprehensive evaluations based on standard metrics—including accuracy, precision, recall, and F1-score—demonstrate that the proposed DMAN significantly outperforms state-of-the-art methods. The system achieved an average accuracy of 96.9% across six distinct activity classes: walking, running, sitting, standing, falling, and empty room scenarios. Furthermore, the system maintains high robustness under challenging conditions such as varying wall materials and sensor misalignments, with a low inference time of 2.8 seconds per sample, making it suitable for real-time applications. This work establishes the DMAN as a scalable and reliable solution for TW-HAR, addressing key limitations of existing methods. Future research directions include exploring additional sensor modalities and enhancing computational efficiency for broader deployment in smart environments and real-time monitoring scenarios. • Introduces a Dynamic Multi-Modal Attention Network (DMAN) for TW-HAR. • Fuses Radar, WiFi, and Acoustic data for robust through-wall detection. • CNN-BiLSTM with dynamic attention enables spatial-temporal learning. • Achieves 96.9% accuracy with low inference time for real-time use. • Scalable solution for healthcare and security monitoring scenarios.