VisHall: Investigating Visual Hallucinations and Evidence Grounding in Vision Language Models
Vision-language models (VLMs) can generate fluent descriptions containing objects, attributes and relationships that are not supported by the visual input, thereby limiting their reliability in trustworthy applications. Existing benchmarks primarily focus on the object-level hallucination detection and often aggregate hallucination into a single evaluation score, providing limited insight into the extent to which generated claims are grounded in visual evidence.To address these limitations, this paper introduces VisHall, a framework for systematically investigating visual hallucination and evidence grounding in vision-language models. The first contribution is a claim-based evaluation framework that decomposes model-generated responses into object, attribute and relationship claims. Each claim is independently evaluated against the image evidence and classified as supported, contradicted or unverifiable. The framework reports hallucination rates alongside claim coverage, enabling a more comprehensive assessment of model reliability.The second contribution is VisHallG, an evidence-grounding approach that requires the model to associate each generated claim with a corresponding image region. A claim is retained only if its associated region provides sufficient visual evidence to support it. Additionally, the framework incorporates discriminative probes with controlled negative sampling to enable more robust and systematic evaluation of hallucination detection capabilities.
Authors
- Tanvi Sree
Publication Details
- Journal
- Figshare
- Published
- 2026-11-15
- DOI
- https://doi.org/10.6084/m9.figshare.35013538
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- preprint