Multimodal Depression Estimation via Contrastive Modality Alignment and Fusion

Video-based multimodal depression estimation is a challenging task that relies on comprehensive video understanding and analysis to predict depression severity. This process involves capturing discriminative information from multimodal features, including auditory, textual, and visual signals in the video data. Existing foundation models primarily focus on fusing multimodal features but often overlook two essential aspects: (1) the alignment of intra- and inter-modality , and (2) preserving modality-specific semantic information while eliminating redundancy . Hence, these issues inevitably result in semantic misalignment between modalities, leading the model to learn redundant or even conflicting representations. To tackle these challenges, we propose a Contrastive Modality Alignment and Fusion (CMAF) framework. Specifically, each modality is divided into multiple segments, and a dynamic normalization strategy is applied to align the temporal context within each modality. Simultaneously, contrastive learning is utilized to achieve semantic alignment across different modalities. Together, these components form the Contextual Modality Alignment (CMA) module. Furthermore, a Cross-Attention Contrastive Fusion (CCF) module is presented, where cross-attention is used to achieve complementarity and enhancement between modalities, while contrastive learning further promotes modal consistency and preserves modality-specific information without redundancy. Therefore, the fused representation can integrate the features of each modality without losing the specific information of the unimodal. Extensive experiments on public datasets, including CMDC, DAIC-WOZ, and E-DAIC, demonstrate that CMAF outperforms advanced depression estimation methods and highlights the essence of effective alignment and fusion.