Don’t Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models

Large Vision Language Models (LVLMs) demonstrate strong capabilities in visual understanding and description, yet often suffer from hallucinations-attributing incorrect or misleading features to images.We observe that LVLMs disproportionately focus on a small subset of image tokens-termed blind tokenswhich are typically irrelevant to the query (e.g., background or non-object regions).We hypothesize that such attention misalignment plays a key role in generating hallucinated responses.To mitigate this issue, we propose Attentional Vision Calibration (AVISC), a test-time approach that dynamically recalibrates the influence of blind tokens without modifying the underlying attention mechanism.AVISC first identifies blind tokens by analyzing layer-wise attention distributions over image tokens, then employs a contrastive decoding strategy to balance the influence of original and blind-tokenbiased logits.Experiments on standard benchmarks, including POPE, MME, and AMBER, demonstrate that AVISC effectively reduces hallucinations in LVLMs.Is there a banana in the image?LLaVA-1.5 InstructBLIP Attention weight distribution Inputs to LVLMs Bounding Box Distribution Attention Map Blind Tokens Inputs to LVLMs Describe this image in detail.Is there a grape in the picture?