COFFEE: Mitigating Hallucination in LVLMs via COllaborative Filtering for Enhanced Eyes

Current Large Vision-Language Models (LVLMs) typically connect a vision encoder and a language model through a vision-language connector. However, hallucination issues in individual modules become more complex after multimodal coupling. Existing training paradigms struggle to address these cross-modal hallucinations effectively, and the lack of interpretability means most solutions rely on correlation-based observations. Previous training-free hallucination mitigation methods lack a fine-grained understanding of the visual perception capabilities of LVLMs and typically perform interventions at the level of the entire model or specific layers, which often leads to greater inference overhead or degraded language generation performance. To address this, we propose COFFEE (COllaborative Filtering For Enhanced Eyes), a training-free, collaborative filtering-based visual enhancement mechanism that provides fine-grained control for enhancing the visual capabilities of LVLMs. It filters out meaningless image tokens on the selected visually-biased attention heads to prevent them from excessively diverting attention away from text tokens after normalization, thereby preserving the model's focus on visual information. Experiments show that COFFEE achieves a good balance between reducing hallucination and maintaining language generation quality. Specifically, Compared to VCD, COFFEE reduces <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$C_{S}$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$C_{I}$</tex> by 36.1 % and 47.0 % respectively, while maintaining a comparable Recall with a slight improvement of 0.1 percentage points.