Multimodal Fusion Network for Human Action Recognition in Industrial Manufacturing

Human action recognition in industrial manufacturing (HAR-IM) plays a crucial role in automating the identification and classification of worker activities. These tasks are inherently fine-grained, requiring the recognition of subtle movements and intricate tool interactions in complex and cluttered environments. While recent advancements in skeleton- and RGB-based recognition methods have demonstrated success in general scenarios, their performance often declines in industrial settings due to challenges such as visually similar actions and significant background noise, which obscure critical motion cues. To address these issues, we propose FusionARM, a multimodal framework that harnesses the complementary strengths of skeleton and RGB modalities. FusionARM leverages skeleton data to prioritize salient spatio-temporal regions within RGB frames, effectively filtering redundant background information and improving the localization of fine-grained activities. The framework introduces a Spatial-Temporal Relevance Selection (STR) mechanism, which aligns keyframes and joint positions to ensure effective fusion of skeleton dynamics with visual context. Additionally, a Model Fusion strategy adaptively balances the contributions of each modality, generating representations that are both discriminative and noise-resilient. Extensive experiments on HAR-IM benchmark datasets validate that FusionARM outperforms state-of-the-art methods, demonstrating its effectiveness in tackling the unique challenges posed by industrial environments.