Information Bottleneck Scores for Identifying Causally Informative Attention Heads in Vision–Language Models

Vision–language models (VLMs) have demonstrated remarkable performance on a wide range of multimodal reasoning tasks, yet their visual grounding mechanisms remain poorly understood and are often unreliable for fine-grained visual concepts. Existing approaches typically rely on raw attention maps or gradient-based saliency, which provide heuristic explanations but lack a causal interpretation of how visual evidence contributes to model predictions. In this paper, we propose an Information Bottleneck Score (IBS) framework that explicitly quantifies the causal importance of visual patches through interventional analysis. By masking candidate image patches and measuring the induced change in the model prediction, the IBS captures patch-level causal contributions rather than correlation-based signals. We further lift patch-level importance to the attention-head level by aggregating the IBS with text-to-image attention, enabling the identification of a small subset of information-transmitting attention heads responsible for visual grounding. Building on the selected heads, we construct refined importance maps that guide visual cropping in a fully training-free manner. Extensive experiments on multiple detail-sensitive benchmarks, including TextVQA, V*, POPE, and DocVQA, demonstrate consistent improvements in fine-grained visual understanding, while evaluations on general-purpose datasets such as GQA, AOKVQA, and VQAv2 confirm that overall reasoning performance is preserved. Additional ablation studies further validate the effectiveness of each component in the proposed framework. Overall, our work provides a causal perspective on visual grounding in VLMs and offers a model-agnostic, training-free approach for both interpreting and enhancing multimodal reasoning.