Dual-Counterfactual Evidence Decomposition and Selective Editing for Hallucination Mitigation in Large Vision-Language Models

Abstract Large vision-language models (LVLMs) often generate fluent but visually unsupported content. Existing training-free methods mainly recalibrate output distributions or edit representations without identifying whether the evidence behind the current claim is visual, linguistic, or shared. This limits intervention precision: unsupported language priors may survive, whereas grounded visual information may be weakened. We introduce a dual-counterfactual evidence decomposition and selective editing framework. Paired visual and linguistic counterfactuals act as sample-specific evidence probes, whose hidden-state responses are decomposed into shared, visual-exclusive, and linguistic-exclusive components. A claim-aware gate attenuates unsupported linguistic evidence only for visually verifiable claims, while an evidence-preserving update protects shared and visual-exclusive support. The method is training-free and requires neither parameter updates nor external verification models. Across three LVLM families, it achieves the highest Accuracy and F1 in all nine POPE backbone--split settings. On CHAIR, it obtains the lowest CHAIR_i on all three evaluated backbones, reducing it by 9.1%-14.7% relative to HulluEdit while maintaining comparable BLEU scores. It also achieves the highest aggregate MME-H on all three backbones, the best results on all four HallusionBench metrics, and a score of 29.3 on MM-Vet. At 23.59 tokens per second, the complete pipeline retains 75.5% of greedy-decoding throughput. These results establish evidence attribution before intervention as an effective and practical strategy for faithful LVLM generation.