Integrating Explainable AI with Retrieval Augmented Generation for Reliable and Personalized Medical Decision Support

The following study explores the integration of Large Language Models (LLMs) into the medical domain by combining Explainable AI (XAI) techniques with Retrieval Augmented Generation (RAG) to deliver reliable and personalized responses that account for the user’s cognitive abilities and background knowledge. One of the key components of the system is the MedRAG toolkit, a Retrieval-Augmented Generation (RAG) framework designed for the medical domain, used for improving response quality by integrating information retrieved from medical documents, which has been extended to consider patient clinical notes as contextual input. This allows the model to provide both predictions and detailed explanations that are based on a combination of patient specific information and external medical knowledge retrieved through the RAG mechanism. The system was validated using the MIMIC QA datasets, a subset of the MIMIC III dataset which contains clinical notes and corresponding reference diagnoses validated by human medical experts, and further tested on the PIMA dataset to predict diabetes. Machine learning models such as Random Forest, Decision Tree, CatBoost, and Neural Network were trained, with explanations generated using SHAP and LIME for both local and global interpretability. The system was evaluated using metrics such as BERT Score, BLEU and ROUGE, demonstrating that combining RAG with XAI improves the model’s performance in providing accurate and understandable medical predictions. This study also highlights that traditional XAI methods may not be sufficient in complex domains like healthcare, where the added context from RAG and LLMs offers significant advantages for interpretability and diagnostic support.