ReSelfVerMM: mitigating hallucination in multimodal LLMs through dataset reconstruction and self-verification

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language tasks. However, their broader adoption remains limited due to the persistent issue of hallucination, particularly object hallucination—where models erroneously predict the presence of non-existent objects. In this study, we identify two primary causes of such hallucinations: (1) insufficient perception of fine-grained objects, and (2) an inherent tendency toward overconfidence, leading models to produce affirmative predictions even in the absence of supporting evidence. To address these challenges, we propose a two-stage framework, ReSelfVerMM, designed to mitigate multimodal hallucinations. In the first stage, we reconstruct the training dataset using structured keyword guidance, enabling the model to better attend to fine-grained objects and thereby enhance visual-textual alignment during training. In the second stage, we introduce a self-verification mechanism that encourages the model to engage in multi-step reasoning, allowing it to reflect upon and validate its own predictions regarding object presence. Experiments conducted on the POPE benchmark demonstrate that ReSelfVerMM effectively reduces hallucination rates while maintaining or even improving overall task performance. This approach offers a scalable solution for enhancing factual consistency in multimodal reasoning and opens new avenues for the design of hallucination-aware MLLMs.