Dual-layer emotion sensing in virtual reality: integrating facial action units and GPT-4o semantic mapping

Introduction Objectively quantifying emotional responses in Virtual Reality (VR) remains a challenging for affective computing and adaptive interface design, particularly during active motor task execution. Methods This study presents a dual-layer emotion sensing framework that combines rule-based Facial Action Unit (AU) analysis with GPT-4 based semantic interpretation to characterize affective responses in immersive environments. As a pilot methodological validation, twenty-one healthy adults performed identical motor tasks in two VR settings (a Contentment Forest and a Horror Morgue) while facial expressions were monitored. Results Co-activations of 63 AUs were classified into six basic emotions; five emotions were retained for statistical analysis, revealing significant differences in emotional distributions between environments (χ 2 (4) = 69.92, p < 0.001). These emotion labels were subsequently reinterpreted into higher-level affective dimensions (Contentment and Anxiety). Discussion The dual-layer framework enables continuous, emotion profiling in VR without intrusive sensors, suggesting potential applications for future affect-aware VR applications in rehabilitation, training, and entertainment.