Multifaceted Assessment and Resolution of Hallucinations in Large Visual-Language Models

The field of Large Visual-Language Models (LVLMs) has made significant strides in integrating visual recognition and language understanding. However, its application in multimodal decision-making and open-ended generation is still constrained by the illusion that generated text does not accurately reflect the visual content. Current benchmarks for assessing LVLMs hallucinations predominantly focus on object hallucinations, thus limiting their scope. In light of this, we constructed the Multifaceted Hallucination Evaluation Bench (MHEB) as a tool to comprehensively assess hallucinations in LVLMs. The MHEB provides an in-depth assessment along four dimensions: Object Detection, Count and Locate, Common Sense Confusion, and Style Adaptation. Our assessment revealed that a model's superior performance on one task does not necessarily indicate its performance on others. In particular, LVLMs demonstrated suboptimal performance on tasks requiring the understanding of common-sense concepts and the identification of fine-grained details. To address the visual hallucination problem, we propose the Visual Contrast Enhancement (VCE), which effectively alleviates hallucinations in LVLMs by comparing original and processed visual information without requiring additional training. Experiments demonstrate that VCE significantly improves the accuracy of visual information.