Higher performance full-body tracking method by integrating multiple tracking techniques based on deep latent space

Real-time full-body tracking in extended reality (XR) environments is crucial for applications such as rehabilitation, sports training, and immersive interactions. However, existing body tracking methods face challenges such as missing data for the lower body, latency, and estimation instability during high-speed movements, making high-precision real-time pose estimation still difficult. In this paper, we propose the Deep Latent Space Assimilation Model (D-LSAM), a novel framework for integrating multiple body tracking techniques in XR environments to achieve more precise, real-time motion capture. Inside-Out Body Tracking (IOBT) on VR headsets can accurately track upper-body and finger movements, yet it struggles to capture areas outside the field of view of the camera—particularly the lower body. On the other hand, external-camera or smartphone-based systems can observe the entire body but often suffer from delays or reduced accuracy. The D-LSAM addresses these limitations by combining a Wasserstein autoencoder for pose compression, a Transformer-driven latent time-stepping module for movement prediction, and a cross-attention gating mechanism that adaptively fuses data from various sources. Experiments using 30 types of motion data collected from a single subject demonstrated that the proposed method consistently outperformed methods based on extended Kalman and deep particle filters in terms of estimation accuracy under both 100-ms and 400-ms delay conditions. In particular, under the 400-ms delay condition, the method achieved high-precision, real-time estimation by reducing the MPJPE to 0.1283 m for sports motions and reducing inference time to 41.8 ms. Future work will emphasize faster inference, improved handling of rapid movements, and support for a wider range of devices. Progress in this methodology holds promise for delivering more immersive XR applications and for advancing fields such as medicine, sports, and rehabilitation.