MSMFT: Multi-Stream Multimodal Factorized Transformer for Human Activity Recognition

Existing video-based human activity recognition (HAR) methods are susceptible to challenges such as lighting variations and occlusions in complex environments. Wearable sensors can effectively mitigate these issues. The fusion of the two can reduce environmental impacts, thus enhancing accuracy. However, the heterogeneity between the two data types presents a challenge for fusion. To address this, we propose an architecture called multi-stream multimodal factorized transformer (MSMFT), which leverages the aggregation capabilities of transformers to achieve alignment and fusion of multimodal features without relying on specific feature extractors. First, we designed a statistical feature projection (SFP) method to effectively tokenize wearable sensor data for transformer models. Second, we introduced temporal embedding (TE) to capture local temporal feature variations. Then, we incorporated a factorized transformer architecture that includes temporal and spatial transformer encoders (STEs) to extract global spatiotemporal features fully. Finally, to avoid the issue of modality entanglement caused by deep fusion, we constructed a multi-stream architecture comprising a video stream, wearable sensors stream, and fusion stream. This design allows for the extraction of both independent and interactive modality features. The performance of the MSMFT method was evaluated on three multimodal datasets: UP-Fall, UTD-MHAD, and MMAct. The results show that MSMFT performs better than the state-of-the-art, with a 16.83% improvement in cross-scene F1-score on the MMAct dataset.