VisHall: Investigating Visual Hallucinations and Evidence Grounding in Vision Language Models

Vision-language models (VLMs) can generate fluent descriptions containing objects, attributes and relationships that are not supported by the visual input, thereby limiting their reliability in trustworthy applications. Existing benchmarks primarily focus on the object-level hallucination detection and often aggregate hallucination into a single evaluation score, providing limited insight into the extent to which generated claims are grounded in visual evidence.To address these limitations, this paper introduces VisHall, a framework for systematically investigating visual hallucination and evidence grounding in vision-language models. The first contribution is a claim-based evaluation framework that decomposes model-generated responses into object, attribute and relationship claims. Each claim is independently evaluated against the image evidence and classified as supported, contradicted or unverifiable. The framework reports hallucination rates alongside claim coverage, enabling a more comprehensive assessment of model reliability.The second contribution is VisHallG, an evidence-grounding approach that requires the model to associate each generated claim with a corresponding image region. A claim is retained only if its associated region provides sufficient visual evidence to support it. Additionally, the framework incorporates discriminative probes with controlled negative sampling to enable more robust and systematic evaluation of hallucination detection capabilities.

Authors

Publication Details

Journal
Figshare
Published
2026-11-15
DOI
https://doi.org/10.6084/m9.figshare.35013538
Primary Topic
Multimodal Machine Learning Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

VisHall: Investigating Visual Hallucinations and Evidence Grounding in Vision Language Models

Tanvi Sree
Figshare
Multimodal Machine Learning Applications
preprint

VisHall: Investigating Visual Hallucinations and Evidence Grounding in Vision Language Models

Tanvi Sree
preprint en

Abstract

Vision-language models (VLMs) can generate fluent descriptions containing objects, attributes and relationships that are not supported by the visual input, thereby limiting their reliability in trustworthy applications. Existing benchmarks primarily focus on the object-level hallucination detection and often aggregate hallucination into a single evaluation score, providing limited insight into the extent to which generated claims are grounded in visual evidence.To address these limitations, this paper introduces VisHall, a framework for systematically investigating visual hallucination and evidence grounding in vision-language models. The first contribution is a claim-based evaluation framework that decomposes model-generated responses into object, attribute and relationship claims. Each claim is independently evaluated against the image evidence and classified as supported, contradicted or unverifiable. The framework reports hallucination rates alongside claim coverage, enabling a more comprehensive assessment of model reliability.The second contribution is VisHallG, an evidence-grounding approach that requires the model to associate each generated claim with a corresponding image region. A claim is retained only if its associated region provides sufficient visual evidence to support it. Additionally, the framework incorporates discriminative probes with controlled negative sampling to enable more robust and systematic evaluation of hallucination detection capabilities.

Figshare
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon