An Explainable Multi-Modal Deep Learning Framework for Investigating the Influences of Dynamic Environmental Exposure on Momentary Stress

Abstract Urban stressors pose critical public health concerns by adversely affecting residents’ mental health. However, past studies have mostly focused on single-modal stressors under static exposure paradigms, with limited exploration of nonlinear environment-stress relationships. This study proposes an explainable multimodal deep learning framework that integrates remote sensing imagery (RSI), street-view imagery (SVI), mobile sensor data, and points of interest (POIs) to accurately predict and explain momentary stress in urban environments. Using 4342 momentary stress records and 125,440 min-level environmental exposure records collected in Hong Kong, we validated the framework, achieving an accuracy of 78.26% and an AUC-ROC of 82.77%. A multilevel gradient-based explainability analysis reveals that (i) environmental exposure within the most recent 25 min exerts the strongest influence on momentary stress and notable spatial heterogeneity exists in stress prediction, highlighting the importance of capturing exposure dynamics; (ii) specific environmental thresholds were identified, including 55 dBA for noise, 25 μg/m3 for PM2.5, 32 °C for temperature, 40%–70% for humidity, and 25% for greenery visibility; and (iii) from an aerial perspective, open spaces consistently suppress stress, while the influence of buildings varies depending on surrounding spatial configuration. This study provides an integrative and explainable tool offering actionable insights for public health interventions.