Gaze and Facial Expression‐Based Multimodal Network for Depression Assessment in Young Adults

ABSTRACT Depression remains a critical mental health challenge worldwide, particularly among young adults. Traditional assessment methods often raise privacy concerns and can heighten psychological resistance. To overcome these barriers, we introduce the Gaze and Facial Expression‐based Multimodal Network (GEM‐Net)—a novel framework engineered to predict depression severity with minimal participant interaction. This approach strategically employs visual stimuli designed to elicit emotional responses and we curated a specialized dataset of 103 young adults (aged 18–24), capturing gaze coordinates, facial video recordings and Beck Depression Inventory‐II (BDI‐II) scores. Contrastive Language‐Image Pre‐Training (CLIP) was utilized to extract facial expression features, while Mamba networks modelled gaze patterns. A dual‐attention mechanism fused both modalities, enhancing cross‐modal interactions for improved accuracy. Evaluated via five‐fold cross‐validation, GEM‐Net achieved a mean absolute error (MAE) of 4.36 and a root mean square error (RMSE) of 5.82, matching state‐of‐the‐art depression severity prediction, and its effectiveness was further validated through ablation experiments. GEM‐Net offers a privacy‐conscious, minimally intrusive and efficient solution for depression assessment among young adults. This integrated multimodal method shows promise for scalable, accessible mental health evaluations, enabling broader clinical and public health implementation.