Error-Guided Multimodal Sample Selection with Hallucination Suppression for LVLMs

Building high-quality multimodal instruction datasets is often time-consuming and costly. Recent studies have shown that a small amount of carefully selected high-quality data can be more effective for improving LVLM performance than large volumes of low-quality data. Based on these observations, we propose an error-guided multimodal sample selection framework with hallucination suppression for LVLM fine-tuning. First, semantic embeddings of queries are clustered to form balanced subsets that preserve task diversity. A visual contrastive decoding module is then used to reduce hallucinations and expose genuinely difficult examples. For closed-ended tasks, such as object detection, we estimate sample value using prediction accuracy; for open-ended question answering, we use the perplexity of generated responses as a difficulty signal. Within each cluster, high-error or high-perplexity samples are preferentially selected to construct a compact yet informative training set. Experiments on the InsPLAD detection benchmark and the PowerQA visual question answering dataset show that our method consistently outperforms random sampling under the same data budget, achieving higher F1, cosine similarity, BLEU (Bilingual Evaluation Understudy), and GPT-4o-based evaluation scores. This demonstrates that hallucination-aware, uncertainty-driven data selection can improve LVLM robustness and data efficiency.