An adaptive spatio-temporal cross-modal learning with directional causal fusion for explainable multimodal depression detection

Depression is one of the most common Psychiatric disorders. Early and accurate diagnosis of depression remains an important clinical need. Recently, multi-modality-based detection strategies have been proposed to promote the early effective diagnosis of depression. Although noteworthy progress has been made in depression detection, existing methods still struggle to learn rich, discriminative feature representations. In addition, current cross-modal fusion methods usually capture the inter-modal interactions symmetrically. Such approaches often overlook directional dependencies between different modalities, which may limit their ability to effectively capture complementary information across modalities. To solve the above problems, the new Adaptive Spatio-Temporal Directional Causal Fusion (AST-DCF) BiLSTM-Mamba framework is proposed for multimodal depression detection. The proposed methodology uses Directional Causal integration (DCF) to determine how one modality might impact or predict another, allowing for more effective integration of auditory and visual data. Additionally, the Spatio-Mamba and Temporal-Mamba-BiLSTM blocks are proposed to learn the spatial and temporal information of the multimodal data. The Temporal-Mamba branch further incorporates BiLSTM with a State Space Model (SSM) to capture fine-grained short-term dependencies and coarse-grained long-term temporal dependencies, yielding more effective and discriminative representations. Additionally, the Cross-Modal Consistency-Guided SHAP-LIME module is proposed, which combines global and local explanations via a cosine-similarity-based consistency mechanism. This improves the interpretability of the suggested framework. Extensive experiments demonstrate that the proposed approach can obtain an 81.86% accuracy, outperforming the baseline approaches. These findings show that the proposed framework can enhance both depression detection performance and model interpretability.