DocVLM-Fingerprint: Document VLM Faithfulness Under Visual Perturbations
DocVLM-Fingerprint studies a narrow failure mode in document visual question answering (VQA):a vision-language model (VLM) can return a plausible short answer while its accompanying claimis unsupported by the annotated evidence. This work implements a reproducible evaluation pipelinefor generated chart, optical character recognition (OCR) document, and scientific figure tasks. Thepipeline includes four visual perturbations, open VLM clients, answer scoring, claim-support labels,plots, and failure records. The main run covers 120 generated examples, 480 perturbed variants, andthree open models. Small clean subsets of ChartQA and DocumentVQA, together with aQwen-only CharXiv perturbation supplement using real charts, provide external checks. Theevidence supports a controlled diagnostic study of failure patterns, not a general ranking of VLMs.
Authors
- Samarth Kukreja (ORCID: https://orcid.org/0009-0008-5494-9138)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24279715
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- preprint