Context-aware retrieval-augmented generation framework for visual question answering in medical imagery

Medical Visual Question Answering (VQA) has underwent significant advancements through the adoption of large vision-language models (VLMs), which typically require fine-tuning for downstream tasks. However, fine-tuning can pose challenges, including high computational demands and potential overfitting risks that can compromise generalisation, especially when training data are limited. In this paper, we introduce an alternative approach termed RAG-MedVQA, a retrieval-augmented framework that integrates retrieval and generation components. Our framework adopts a modular design that decouples the retrieval and generation components, enabling flexible integration with a variety of model choices. In the retrieval phase, we leverage the BioMedCLIP retriever to compute dense image embeddings and retrieve the most similar images from the training set to the query image along with their corresponding diagnostic dialogues. In the generation phase, the query image, its associated question, and the retrieved context are processed by Qwen2VL-7B, a multi-image capable vision-language model that fuses clinical content from the exemplars to synthesise the final answer. More specifically, we form a structured prompt to guide Qwen2VL-7B effectively during response generation. The experimental results show that RAG-MedVQA achieves competitive performance compared to conventional fine-tuning approaches.

Authors

Institutions

Publication Details

Journal
Connection Science
Published
2026-11-12
DOI
https://doi.org/10.1080/09540091.2026.2761838
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Context-aware retrieval-augmented generation framework for visual question answering in medical imagery

Mansour Zuair, Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Luluah Alhusain
Connection Science
Multimodal Machine Learning Applications
article

Context-aware retrieval-augmented generation framework for visual question answering in medical imagery

Mansour Zuair, Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Luluah Alhusain
article en

Abstract

Medical Visual Question Answering (VQA) has underwent significant advancements through the adoption of large vision-language models (VLMs), which typically require fine-tuning for downstream tasks. However, fine-tuning can pose challenges, including high computational demands and potential overfitting risks that can compromise generalisation, especially when training data are limited. In this paper, we introduce an alternative approach termed RAG-MedVQA, a retrieval-augmented framework that integrates retrieval and generation components. Our framework adopts a modular design that decouples the retrieval and generation components, enabling flexible integration with a variety of model choices. In the retrieval phase, we leverage the BioMedCLIP retriever to compute dense image embeddings and retrieve the most similar images from the training set to the query image along with their corresponding diagnostic dialogues. In the generation phase, the query image, its associated question, and the retrieved context are processed by Qwen2VL-7B, a multi-image capable vision-language model that fuses clinical content from the exemplars to synthesise the final answer. More specifically, we form a structured prompt to guide Qwen2VL-7B effectively during response generation. The experimental results show that RAG-MedVQA achieves competitive performance compared to conventional fine-tuning approaches.

Connection ScienceVol. 38(1)
King Saud University (SA)
King Saud University
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon