Text-Calibrated Visual Value Residuals for Contrastive Decoding in Image Captioning

We fit a predictor of attention values from keys on text and apply it to image tokens in LLaVA-1.5-7B. Version 1 contrasts predictions from the original and predicted visual values at every token. On an independent 500-image set, it reduces sentence CHAIR from 50.20% to 45.00% and instance CHAIR from 14.96% to 11.71% relative to matched greedy decoding. Version 2 selects its first three tokens by ordinary greedy decoding. If a fourth token is needed, it extrapolates the image values away from their predictions and continues on one branch. Across a separate 1,000-image set, sentence CHAIR is 34.60%, instance CHAIR is 9.849% and category recall is 80.81%. All distinct POPE and MME inputs evaluated with the binary stopping rule end at the first token.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-20
DOI
https://doi.org/10.5281/zenodo.24390374
Primary Topic
Multimodal Machine Learning Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Text-Calibrated Visual Value Residuals for Contrastive Decoding in Image Captioning

Ziheng Li
4 links
Zenodo (CERN European Organization for Nuclear Research)
Multimodal Machine Learning Applications
preprint

Text-Calibrated Visual Value Residuals for Contrastive Decoding in Image Captioning

Ziheng Li
preprint en

Abstract

We fit a predictor of attention values from keys on text and apply it to image tokens in LLaVA-1.5-7B. Version 1 contrasts predictions from the original and predicted visual values at every token. On an independent 500-image set, it reduces sentence CHAIR from 50.20% to 45.00% and instance CHAIR from 14.96% to 11.71% relative to matched greedy decoding. Version 2 selects its first three tokens by ordinary greedy decoding. If a fourth token is needed, it extrapolates the image values away from their predictions and continues on one branch. Across a separate 1,000-image set, sentence CHAIR is 34.60%, instance CHAIR is 9.849% and category recall is 80.81%. All distinct POPE and MME inputs evaluated with the binary stopping rule end at the first token.

Zenodo (CERN European Organization for Nuclear Research)
Huazhong University of Science and Technology (CN)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon