A Deep Learning Framework on Spatio-Temporal and Multi-Modal Features of the Video for Effective Facial Expressions and Stress Inference

Current vision-based methods for stress detection rely on static facial expressions analysis and suffer from the problems of reliability, accuracy, and generalization. This study proposes a deep learning model for real-time stress detection based on multimodal spatio-temporal structural fusion. Our model utilizes a dual-stream CNN for obtaining frame spatial features and sequence temporal features. To complement this data, we also estimate heart rate variability (HRV) from remote photoplethysmography (rPPG) extracted from the facial video. Tested on the combined datasets of FER-2013, AffectNet, and DKEFS, the system identifies expressions with 88.5% accuracy and achieves approximately 25% better precision in stress inference than baseline single-mode CNN models. The system is accelerated by TensorRT to run in real time at over 30 FPS on a consumer-grade GPU.