Text-Calibrated Visual Value Residuals for Contrastive Decoding in Image Captioning
We fit a predictor of attention values from keys on text and apply it to image tokens in LLaVA-1.5-7B. Version 1 contrasts predictions from the original and predicted visual values at every token. On an independent 500-image set, it reduces sentence CHAIR from 50.20% to 45.00% and instance CHAIR from 14.96% to 11.71% relative to matched greedy decoding. Version 2 selects its first three tokens by ordinary greedy decoding. If a fourth token is needed, it extrapolates the image values away from their predictions and continues on one branch. Across a separate 1,000-image set, sentence CHAIR is 34.60%, instance CHAIR is 9.849% and category recall is 80.81%. All distinct POPE and MME inputs evaluated with the binary stopping rule end at the first token.
Authors
- Ziheng Li
Institutions
- Huazhong University of Science and Technology (CN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-20
- DOI
- https://doi.org/10.5281/zenodo.24390374
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- preprint