Concept-wise Feature Sampling: Mitigating Object Hallucinations for Large Vision-Language Models

Large Vision-Language Models (LVLMs) have advanced significantly due to the integration of multimodal data, enabling improved performance across various applications. However, the persistent challenge of hallucination-where models generate plausible but inaccurate content-remains a significant barrier to their practical deployment, particularly in critical fields such as healthcare and autonomous driving. This paper presents a concept-wise feature sampling (CFS) framework that addresses object hallucination by leveraging the strengths of existing pixel-level and feature-level approaches. By utilizing CLIP components to identify relevant object concepts and employing a semantic segmentation model (Mask CLIP) to sample least similar features, our method enhances the visual interpretability of predictions while maintaining computational efficiency. Rigorous evaluations on standard benchmarks show that CFS effectively reduces object hallucinations in LVLMs and supports a more explainable flow of suboptimal multimodal information, paving the way for safer applications of these models.