Multimodal Healthcare in the Metaverse: Integrating Multimedia Information Retrieval, Physiological Sensing, and Markerless Kinematic Analytics Within the CareSync Framework

Recent advances in immersive computing and artificial intelligence have enabled the development of adaptive, human-centered digital healthcare systems. This paper presents CareSync, an emotion-aware, data-driven architecture for personalized interventions and Multimedia Information Retrieval within the Healthcare Metaverse. Built on the SenseCare KM-EP platform, the CareSync framework integrates distributed healthcare services to orchestrate real-time, multi-source data processing, multimodal feature extraction, and semantic graph generation. This work advances T-Rehab, the platform’s metaverse-based telerehabilitation subsystem, by proposing a workflow for integrating multimedia information retrieval (MIR), physiological sensing, and markerless kinematic analytics within a metaverse-based healthcare environment, and by implementing this workflow as a functional real-time system. The implementation integrates camera-based physiological sensing, including POS-based remote photoplethysmography (rPPG) heart rate estimation with bandpass autocorrelation, chest-motion respiratory rate estimation with RIIV cross-validation, and RMSSD-based stress heuristics, with markerless biomechanical analysis. Using MediaPipe and Leap Motion tracking, the platform further assesses facial emotions, pain-related behavioral cues, and full-body kinematics, including joint angles, range of motion, movement symmetry, and repetition counting, enabling detailed real-time monitoring during immersive telerehabilitation sessions. For data indexing and retrieval, structured session outcome summaries and co-occurrence graphs are generated from telemetry and vitals, supporting both exact filter queries and graph-similarity ranking for sessions preserved via non-video 3D avatar replays. Actionable information is surfaced through real-time linear-trend and threshold-based alerts that monitor system-estimated indicators of participant stability, safety, and affect. The updated T-Rehab was evaluated on the (N = 26) participant cohort from the rPPG-10 benchmark dataset (27 subjects in total, with Subject 4 excluded). The evaluation compared five rPPG methods, GREEN, CHROM, POS, PCA, and ICA, using synchronized 30-s windows and the ECG reference. The results identified GREEN as the best-performing method, achieving the lowest mean absolute error (MAE) of 17.74 BPM. Furthermore, the participant-level evaluation and consistent 30-s windowing provide a transparent and reproducible basis for comparing non-contact heart-rate estimation methods. The results are intended as a methodological benchmark and should not be interpreted as demonstrating clinical accuracy.