A Multimodal Framework for Integrating Motion Capture Monitoring into ASD Screening Utilizing XR Applications

Full-body tracking (FBT) enhances embodiment and motion-based interaction in extended reality (XR), particularly for virtual reality (VR) applications such as those used for autism spectrum disorder (ASD) screening. While Meta Movement Software Development Kit (SDK) provides eye and hand tracking, its estimation-based FBT skeleton conflicts with Open Sound Control (OSC)-based body tracking data, leading to body proportion distortion and misalignment when external trackers like SlimeVR are used. To resolve this, we adopt the Virtual Motion Capture (VMC) protocol skeleton as the primary FBT framework, integrating OSC-based body tracking with OpenXR for head tracking and Meta’s SDK for eye and hand tracking. Multiple custom OSC messages are created by retrieving positional and rotational data from custom input actions, allowing real-time player headspace detection and precise avatar alignment. This approach ensures accurate skeletal proportions and real-time avatar alignment, eliminating reliance on Meta’s prediction-based body tracking. By leveraging custom OSC messages and SlimeVR’s body proportion calibration, the proposed system offers a more stable and precise FBT solution for standalone VR applications.