AI-Based Multimodal Data Fusion for Psychological Stress Assessment and Cardiovascular Risk Early Warning among University Students in the Greater Bay Area

Abstract Background The acceleration of integration within the Greater Bay Area has presented unique mental health challenges for university students, stemming from differences in educational systems, languages, and cultures. There is an urgent need for culturally adaptive mental health screening tools. Existing large models, while achieving high accuracy in English contexts, lack validation in non-English environments and fail to account for regional stressors. Traditional mental health services, often relying on unimodal assessments such as standardized scales and professional interviews, demonstrate insufficient cultural sensitivity and can have high false-positive rates. Purpose Accumulating evidence suggests that prolonged psychological stress among young adults is closely associated with dysregulation of the autonomic nervous system and elevated cardiovascular risk, highlighting the necessity of integrating mental health assessment with early cardiovascular risk screening. This situation urgently calls for innovative, multimodal solutions capable of providing more accurate and culturally attuned mental health and physiological risk assessment for this cross-cultural student population. Methods To address this gap, this study constructed a BERT–LLaMA-based multimodal large model framework for university student mental health assessment. The research focused on a large sample of 120,000 university students from six higher education institutions across the cross-cultural Greater Bay Area. The hybrid model was trained on a dataset of 12,000 multimodal data points, integrating text, speech, and physiological data derived from wearable ECG signals to assess mental health status and stress-related physiological responses. The inclusion of ECG-based features enabled the extraction of heart rate variability (HRV) indicators associated with stress regulation and cardiovascular function, allowing the model to simultaneously capture psychological distress and potential cardiovascular risk signals. Among the participants, 18,500 provided speech samples, and 10,200 participated in ECG data collection, forming a large-scale, heterogeneous dataset. A stratified sampling approach was used to select 2,000 students for clinical interviews conducted by psychologists using standard assessments, which served as a baseline for validating the model’s performance in real-world settings. Results The study’s results demonstrated four main innovations. First, the AI-driven multimodal mental health assessment framework was validated using large-scale empirical evidence from cross-regional higher education institutions. Second, multimodal fusion significantly outperformed single-modality screening approaches. Third, physiological signals derived from wearable ECG data played a critical role in distinguishing transient stress responses from more persistent psychological disorders, offering early indicators of autonomic imbalance and potential cardiovascular vulnerability. Additionally, vocal features demonstrated strong sensitivity in anxiety detection, supporting modality-specific contributions to mental health assessment. Conclusions This study successfully constructed and validated a multimodal AI framework tailored for the Greater Bay Area, optimizing cross-cultural mental health services for university students through culturally embedded and physiologically informed solutions. The findings confirm that multimodal fusion approaches surpass single-modality methods in both accuracy and clinical relevance. By integrating psychological indicators with cardiovascular-related physiological signals, the framework provides a novel pathway for early identification of stress-related cardiovascular risk in young adults. Future research should emphasize longitudinal follow-up, enhanced model interpretability, and ethical governance as AI-driven mental health and cardiovascular screening tools are increasingly adopted.