RAG-Guided Large Language Models for Visual Spatial Description with Adaptive Hallucination Corrector
Visual Spatial Description (VSD) is an emerging image-to-text task which aims at generating descriptions of the spatial relationships between given objects in an image. In this paper, we apply Retrieval-Augmented Generation (RAG) technology in guiding Multimodal Large Language Models (MLLMs) for the task of VSD, complemented by an Adaptive Hallucination Corrector, and further fine-tuning them to bolster semantic understanding and overall model efficacy. We found that our approach demonstrated higher accuracy and fewer hallucination errors in both spatial relationship classification and visual language description tasks within the VSD task, achieving state-of-the-art results.
Authors
- Jiaen Liang (ORCID: https://orcid.org/0009-0001-8309-1301)
- Zerui Zhang
- Jianqing Sun (ORCID: https://orcid.org/0009-0007-3598-8564)
- Jun Yu (ORCID: https://orcid.org/0000-0002-3197-8103)
- Gongpeng Zhao (ORCID: https://orcid.org/0009-0004-1019-1477)
- Fanrui Zhang (ORCID: https://orcid.org/0000-0002-1078-430X)
- Yunxiang Zhang
- Yang Zhao (ORCID: https://orcid.org/0009-0007-2692-9458)
- Fengzhao Sun
- Yaohui Zhang (ORCID: https://orcid.org/0009-0009-0147-5335)
- Qingsong Liu (ORCID: https://orcid.org/0009-0004-1208-3031)
Institutions
- University of Science and Technology of China (CN)
- Xi'an Jiaotong University (CN)
- University of Science and Technology Beijing (CN)
Publication Details
- Published
- 2024-10-26
- DOI
- https://doi.org/10.1145/3664647.3688990
- Citations
- 7
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 1.74