Multimodal Knowledge-Infused VLM for Respiratory Disease Prediction and Clinical Report Generation

Accurate and interpretable computer-aided diagnosis (CAD) for respiratory diseases remains a significant challenge due to the complex multimodal nature of medical data. Existing systems often fall short by focusing on unimodal retrieval analysis or lack sophisticated retrieval strategies, which limit both diagnostic reliability and interpretability. To address this, we propose a novel end-to-end CAD framework that employs a large language model (LLM) with advanced retrieval-augmented generation (RAG) techniques, leveraging multimodal retrieval and structured radiology report generation. The framework employs a dedicated vision encoder for binary identifying view types (frontal anterior-posterior (AP), Lateral View (LV)) and multi-label classification for 14 disease categories, with Grad-CAM providing visual interpretability. Concurrently, an advanced text encoder processes unstructured medical reports, enabling the generation of structured zero-shot reports. We propose three novel RAG approaches to enhance retrieval accuracy and reduce LLM hallucination by improving contextual alignment for medical report generation: (1) Caption-Augmented Fusion (CAF), (2) CLIP-Aligned Fusion (CLAF), and (3) Domain-Grounded Fusion (DGF). The performance of our radiology report generation (RRG) is rigorously evaluated using a comprehensive suite of lexical, semantic, and clinical metrics, including METEOR, ROUGE-L, BERTScore, and CheXpert Labeler. The clinical relevance and diagnostic accuracy are further validated by a panel of expert radiologists, establishing a gold standard. Our proposed CAD system can accurately diagnose respiratory conditions and generate structured radiology reports within 21.39 seconds. This rapid and accurate system holds considerable promise for real-world clinical applications and is well-suited to meet the demands of modern medical practice.