3D human pose estimation in the sailing simulator based on somatosensory interaction

Abstract The increasing research on sailing simulators has facilitated the advancement and widespread adoption of land-based sailing training. The somatosensory interaction offers benefits such as ease of deployment, low cost, and natural interactivity. This paper introduces a sailing simulator based on somatosensory interaction, wherein a standard camera and 3D human pose estimation (3DHPE) technology are utilized for sailing simulation. The aim of the study is to strike a balance between interactivity and deployment complexity. Despite the recent advances in 3DHPE, its use in real-time deployment scenarios has not been fully explored. The use of 3DHPE for pose estimation of dynamic users, wherein achieving a balance between real-time efficiency and accuracy is challenging. To address this, we propose a 3D pose estimation approach that integrates a graph-guided state space (STGJMamer). By using the lightweight transformer model PoseformerV2 as a baseline, the proposed method yields good real-time efficiency while integrating spatiotemporal extended graph convolutions and hierarchical joint enhancement Mamba. Furthermore, the model can efficiently capture both global and local features, and this ultimately enhances the pose estimation accuracy for dynamic users. The experimental results demonstrate that our approach achieves a frame-wise inference speed of 52 FPS, satisfying real-time constraints. Furthermore, it achieves an average mean per-joint position error (MPJPE) of 29.5 mm on the MPI-INF-3DHP dataset, outperforming most existing methods. Finally, we deploy the STGJMamer in a somatosensory interaction-based sailing simulator system and study its feasibility in real-world applications. The code is available at https://gitee.com/chemingmei/stgjmamer .