Research on action recognition method based on multimodal fusion of HRNet, IMU, and Kinect

The demand for high-precision and low-latency action recognition in intelligent human-computer interaction is urgent. Aiming at the single-modal defects of HRNet visual recognition environmental sensitivity, IMU cumulative error, Kinect high latency, and the insufficient dynamic adaptation of traditional fusion, this study proposes an action-scene dual-driven dynamic weighting multimodal fusion strategy. The AS-DDW framework integrates HRNet, IMU and Kinect data, uses CNN-LSTM model to realize joint perception of action and scene, and dynamically adjusts modal weights combined with spatiotemporal alignment technology. Experiments show that the fusion scheme has key point positioning error of 1.63mm, joint angle error of 2.89°, system latency of 24.6ms, and action classification accuracy of 94.1% in complex scenes, providing a solution for multimodal action recognition that balances accuracy and real-time performance.