Multimodal Sentiment-Symptom Disentanglement network for interpretable mental health classification from text and visual data

Mental health disorders manifest through intertwined affective and symptomatic signals that are difficult to separate using unimodal or naively fused multimodal models, often producing classifiers that conflate general sentiment polarity with clinically meaningful symptom markers and offer little insight into their own decision process. This paper proposes a Multimodal Sentiment-Symptom Disentanglement Network (MSSD-Net) that explicitly decomposes textual and visual representations into two orthogonal latent subspaces — a sentiment-affect subspace capturing general emotional tone and a symptom-specific subspace capturing clinically relevant behavioral and linguistic markers (e.g., psychomotor cues, facial action units, negation and rumination patterns in text). Modality-specific transformer encoders first extract joint text and visual embeddings, which are passed through a disentanglement module trained with a combination of mutual-information minimization, orthogonality regularization, and adversarial subspace classifiers to enforce separation between the two latent streams. A cross-modal attention fusion layer then recombines the disentangled symptom subspaces from both modalities for downstream mental health classification (e.g., depression, anxiety severity levels). At the same time, the sentiment subspace is retained only as an auxiliary, interpretability-supporting signal. To ensure clinical transparency, the framework incorporates a post-hoc and intrinsic explainability layer that projects attention weights and subspace activations back onto input tokens and facial regions, yielding symptom-attributed saliency maps rather than sentiment-confounded heatmaps. Experiments on benchmark multimodal mental health datasets demonstrate that MSSD-Net achieves 89.4% F1-score over E-DAIC and state-of-the-art multimodal baselines, while substantially improving the faithfulness and clinician-rated interpretability of the generated explanations. Ablation studies reveal that disentangling sentiment and symptom is responsible for the performance gain and improved interpretability in joint entangled fusion. The findings indicate that disentangled multimodal architectures can serve as promising, clinically sound methods for automated mental health screening from naturalistic, real-world data.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-16
DOI
https://doi.org/10.1038/s41598-026-75097-9
Primary Topic
Mental Health via Writing
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Multimodal Sentiment-Symptom Disentanglement network for interpretable mental health classification from text and visual data

Sasanko Sekhar Gantayat, K Sunil Manohar Reddy
1 links
Scientific Reports
Mental Health via Writing
article

Multimodal Sentiment-Symptom Disentanglement network for interpretable mental health classification from text and visual data

Sasanko Sekhar Gantayat, K Sunil Manohar Reddy
article en

Abstract

Mental health disorders manifest through intertwined affective and symptomatic signals that are difficult to separate using unimodal or naively fused multimodal models, often producing classifiers that conflate general sentiment polarity with clinically meaningful symptom markers and offer little insight into their own decision process. This paper proposes a Multimodal Sentiment-Symptom Disentanglement Network (MSSD-Net) that explicitly decomposes textual and visual representations into two orthogonal latent subspaces — a sentiment-affect subspace capturing general emotional tone and a symptom-specific subspace capturing clinically relevant behavioral and linguistic markers (e.g., psychomotor cues, facial action units, negation and rumination patterns in text). Modality-specific transformer encoders first extract joint text and visual embeddings, which are passed through a disentanglement module trained with a combination of mutual-information minimization, orthogonality regularization, and adversarial subspace classifiers to enforce separation between the two latent streams. A cross-modal attention fusion layer then recombines the disentangled symptom subspaces from both modalities for downstream mental health classification (e.g., depression, anxiety severity levels). At the same time, the sentiment subspace is retained only as an auxiliary, interpretability-supporting signal. To ensure clinical transparency, the framework incorporates a post-hoc and intrinsic explainability layer that projects attention weights and subspace activations back onto input tokens and facial regions, yielding symptom-attributed saliency maps rather than sentiment-confounded heatmaps. Experiments on benchmark multimodal mental health datasets demonstrate that MSSD-Net achieves 89.4% F1-score over E-DAIC and state-of-the-art multimodal baselines, while substantially improving the faithfulness and clinician-rated interpretability of the generated explanations. Ablation studies reveal that disentangling sentiment and symptom is responsible for the performance gain and improved interpretability in joint entangled fusion. The findings indicate that disentangled multimodal architectures can serve as promising, clinically sound methods for automated mental health screening from naturalistic, real-world data.

Scientific Reports
SR University (IN)
Good health and well-being
Mental Health via Writing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon