Multimodal Fusion of Wearable and Vision Data: Exploring Early and Model Fusion for Human Activity Understanding

Human Activity Recognition (HAR) has become increasingly important in physiotherapy, where accurate monitoring of patient movements supports rehabilitation and recovery. Traditional HAR systems often rely on single-modality data, limiting their ability to capture complex body movements and subtle variations in posture. To address these limitations, we introduce MSA-MURD, a novel multimodal dataset combining wearable inertial sensor (IMU) and vision-based skeleton data, collected under expert physical therapy supervision. This dataset provides a foundation for evaluating multimodal HAR in upper-body rehabilitation exercises. A comparative study of fusion strategies was presented, focusing on early fusion and model-level (intermediate) fusion techniques, and analyzing their impact on activity recognition performance. Additionally, we implement an ensemble learning approach to further enhance model robustness and generalization. Experimental results demonstrate that intermediate fusion consistently outperforms early fusion, and ensemble models achieve superior recognition accuracy. These findings underscore the value of integrating multimodal data and ensemble learning for improving HAR performance, particularly in complex rehabilitation scenarios. Our work provides practical insights for designing intelligent rehabilitation monitoring systems and contributes a publicly available resource for advancing multimodal HAR research.