From action units to grounded explanations: Multiple Instance Learning and Vision-Language Reasoning for video FER

Facial expressions alone are often insufficient to infer emotion: human observers routinely use scene, body and social context to disambiguate facial cues. Recent multimodal frameworks built on vision-language and large language models (VLMs/LLMs) reason jointly over facial and contextual evidence, but they typically process entire videos. This is computationally costly, and the few emotionally expressive frames are diluted among many uninformative ones. We introduce Reasoning from Facial Evidence for Emotion Recognition (ReFER), which focuses multimodal reasoning on the most relevant moments. A Multiple Instance Learning (MIL) module predicts the emotion from an Action Unit (AU) sequence and scores each frame's relevance. A VLM then describes the visual context of the top-scoring frames. Finally, an instruction-tuned LLM fuses the MIL prediction, the AU measurements and the context description into a final label and a natural-language explanation. Beyond accuracy, we analyze which inputs actually drive the fused prediction. Across four in-the-wild video facial expression recognition (FER) benchmarks and five open-weight fusion LLMs, the best fusion models outperform the AU-only MIL baseline on seven of eight dataset-metric pairs, with the largest gains on AFEW and DFEW. Component ablations show that the initial MIL prediction, rather than the contextual description, is the dominant input to the fusion stage. A label-corruption analysis further reveals that all fusion models anchor strongly on the supplied initial prediction, adopting corrupted labels in 82.6%–99.9% of cases. These results show that an explanation consistent with the available evidence does not imply that the evidence faithfully determines the prediction and points to the need for fusion mechanisms explicitly designed to reconcile conflicting multimodal evidence.

Authors

Institutions

Publication Details

Journal
Frontiers in Artificial Intelligence
Published
2026-11-17
DOI
https://doi.org/10.3389/frai.2026.1984257
Primary Topic
Emotion and Mood Recognition
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

From action units to grounded explanations: Multiple Instance Learning and Vision-Language Reasoning for video FER

Vittorio Cuculo, Rita Cucchiara, Livia Del Gaudio
1 links
Frontiers in Artificial Intelligence
Emotion and Mood Recognition
article

From action units to grounded explanations: Multiple Instance Learning and Vision-Language Reasoning for video FER

Vittorio Cuculo, Rita Cucchiara, Livia Del Gaudio
article en

Abstract

Facial expressions alone are often insufficient to infer emotion: human observers routinely use scene, body and social context to disambiguate facial cues. Recent multimodal frameworks built on vision-language and large language models (VLMs/LLMs) reason jointly over facial and contextual evidence, but they typically process entire videos. This is computationally costly, and the few emotionally expressive frames are diluted among many uninformative ones. We introduce Reasoning from Facial Evidence for Emotion Recognition (ReFER), which focuses multimodal reasoning on the most relevant moments. A Multiple Instance Learning (MIL) module predicts the emotion from an Action Unit (AU) sequence and scores each frame's relevance. A VLM then describes the visual context of the top-scoring frames. Finally, an instruction-tuned LLM fuses the MIL prediction, the AU measurements and the context description into a final label and a natural-language explanation. Beyond accuracy, we analyze which inputs actually drive the fused prediction. Across four in-the-wild video facial expression recognition (FER) benchmarks and five open-weight fusion LLMs, the best fusion models outperform the AU-only MIL baseline on seven of eight dataset-metric pairs, with the largest gains on AFEW and DFEW. Component ablations show that the initial MIL prediction, rather than the contextual description, is the dominant input to the fusion stage. A label-corruption analysis further reveals that all fusion models anchor strongly on the supplied initial prediction, adopting corrupted labels in 82.6%–99.9% of cases. These results show that an explanation consistent with the available evidence does not imply that the evidence faithfully determines the prediction and points to the need for fusion mechanisms explicitly designed to reconcile conflicting multimodal evidence.

Frontiers in Artificial IntelligenceVol. 9
University of Modena and Reggio Emilia (IT)
European High Performance Computing Joint Undertaking
Emotion and Mood Recognition
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon