Multimodal fusion of inertial sensors and single RGB camera data for 3D human pose estimation based on a hybrid LSTM-Random forest fusion network

Over the last decades, the capturing and monitoring of human motions and poses have found increasing interest, with the recent advancements in interconnected sensors and computer vision applications. By effectively fusing data from inertial sensors and visual cues, the limitations that each type of data introduces can be overcome leading to highly accurate human pose estimation. However, in such applications, a complex setup with many multiview cameras and inertial sensors is required for the systems to perform effectively. This work proposes a multimodal fusion system based on a single RGB camera and six inertial sensors for effective human pose estimation. For the unimodal data analysis, two state-of-the-art methods have been adopted; namely the Transformer Inertial Poser network for the inertial sensor data analysis and the MediaPipe’s 3D Pose Landmarker for the analysis of the camera data. 3D joint coordinate results from both unimodal analysis methods are fused in a decision-level fusion approach using a hybrid model architecture combining a deep learning network and a machine learning regression model. Two different deep learning network architectures were tested; a fully connected network architecture and a Long Short-Term Memory based network architecture, followed by a Random Forest regression model as the last layer of our hybrid model. Results on the TotalCapture dataset reveal that our method outperforms other state-of-the-art systems, reducing the mean per joint position error by 13.9 mm, while using a simpler data acquisition setup with only a single RGB camera and six inertial sensors. • Multimodal 3D human pose estimation with inertial sensors and single RGB camera data • Novel hybrid decision-level fusion network merging deep learning and machine learning • Superiority of the proposed hybrid architecture over simple deep learning models • Beyond state-of-the-art results reducing the mean per joint position error by 13.9 mm • Simple setup of 6 inertial sensors and a single RGB camera for 3D pose estimation.