Exploring multimodal, non-invasive stress assessment through audio-visual and textual cues integrated with psychometric survey data

Stress is a widespread psychological concern that often manifests alongside conditions such as anxiety and depression. Traditional self-report tools like the Perceived Stress Scale (PSS-10) may not fully capture an individual’s stress experience. This study explores whether integrating multimodal biometric data through video, audio, and transcriptions can enhance stress detection by providing a more comprehensive and interpretive point of view. Participants completed the PSS-10 while being recorded, and emotional features were extracted using machine learning models across the three biometric modalities. Results revealed weak correlations among the modalities, indicating that each captures distinct aspects of stress. Notably, the combined biometric score demonstrated greater sensitivity than the PSS-10 alone, suggesting that multimodal models may detect stress-related states that self-reports overlook. These findings support the development of more comprehensive stress assessment tools, although they are not intended to replace professional clinical evaluation. • Stress analyzed as a multidimensional, contextual, and cumulative construct. • Emotional cues from text, audio, and visuals proposed as objective indicators. • Multimodal ML framework integrates emotional and psychometric stress data. • Think-aloud techniques enhance accessibility of stress self-assessment tools. • Multimodal fusion improves stress classification over unimodal baselines.