GCR-TE: Grounded generative counterfactual rewriting for text enhanced multimodal sentiment analysis

Real-world multimodal sentiment analysis often suffers from a paradoxical bottleneck: text is the dominant signal carrier, yet it is also the easiest channel to be hijacked by spurious co-occurrences that trigger shortcut learning and lead to severe out-of-distribution degradation. We propose GCR-TE, a text-enhancement framework for multimodal sentiment learning that combines multimodal grounding with generative counterfactual rewriting. First, GCR-TE builds sentiment evidence anchors from fused audio-visual-text representations and attribution-based consistency scores, separating emotion-bearing spans from suspicious co-occurrence spans in the transcript. Second, a grounded counterfactual rewriter generates a family of label-preserving textual variants by controllably substituting, paraphrasing, and style-perturbing only the suspicious spans while keeping the anchored sentiment cues and their semantic roles intact. Finally, we train the model with a triple objective: (i) counterfactual invariance to suppress shortcut features, (ii) anchor-focused attention gain to selectively amplify reliable sentiment cues, and (iii) cross-modal alignment regularization to ensure semantic consistency between augmented text and the original audio-visual evidence. Without requiring extra human annotations, GCR-TE simultaneously improves robustness against spurious correlations and enhances the effective utilization of sentiment-bearing words, offering a controllable and interpretable text-enhanced paradigm for multimodal sentiment analysis.

Authors

Institutions

Publication Details

Journal
Array
Published
2026-04-22
DOI
https://doi.org/10.1016/j.array.2026.100849
Primary Topic
Sentiment Analysis and Opinion Mining
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

GCR-TE: Grounded generative counterfactual rewriting for text enhanced multimodal sentiment analysis

Shibo Ji, Bo Yang
Array
Sentiment Analysis and Opinion Mining
article

GCR-TE: Grounded generative counterfactual rewriting for text enhanced multimodal sentiment analysis

Shibo Ji, Bo Yang
article en

Abstract

Real-world multimodal sentiment analysis often suffers from a paradoxical bottleneck: text is the dominant signal carrier, yet it is also the easiest channel to be hijacked by spurious co-occurrences that trigger shortcut learning and lead to severe out-of-distribution degradation. We propose GCR-TE, a text-enhancement framework for multimodal sentiment learning that combines multimodal grounding with generative counterfactual rewriting. First, GCR-TE builds sentiment evidence anchors from fused audio-visual-text representations and attribution-based consistency scores, separating emotion-bearing spans from suspicious co-occurrence spans in the transcript. Second, a grounded counterfactual rewriter generates a family of label-preserving textual variants by controllably substituting, paraphrasing, and style-perturbing only the suspicious spans while keeping the anchored sentiment cues and their semantic roles intact. Finally, we train the model with a triple objective: (i) counterfactual invariance to suppress shortcut features, (ii) anchor-focused attention gain to selectively amplify reliable sentiment cues, and (iii) cross-modal alignment regularization to ensure semantic consistency between augmented text and the original audio-visual evidence. Without requiring extra human annotations, GCR-TE simultaneously improves robustness against spurious correlations and enhances the effective utilization of sentiment-bearing words, offering a controllable and interpretable text-enhanced paradigm for multimodal sentiment analysis.

ArrayVol. 30
Bowie State University (US)
Openalex Percentile: Top 18%
Sentiment Analysis and Opinion Mining
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.