Deep Learning-Based Multimodal Fusion for Heart Rate and Stress Estimation using remote PPG

This paper presents a novel smartphone-based multimodal system for heart rate (HR) and stress estimation using remote photoplethysmography (rPPG), thermal imaging, and voice stress analysis. In contrast to existing datasets like StressID and VicarPPG-2, our method employs decision-level fusion to obtain a stress classification accuracy of 92.1, outperforming single-modality models (RGB: 78.2, Thermal: 81.5). We provide the first publicly available dataset with synchronized RGB-thermal-voice data and physiological ground truth (ECG/SpO2), collected during controlled stress tasks (cognitive, social, emotional). Our experiments show that rPPG achieves a mean absolute error (MAE) of 2.5 BPM under bright light and 6.1 BPM under movement. This work enables real-time, non-invasive stress monitoring for mental health and workplace wellness."