Explainable Medical Image Captioning

In the age of current information superhighways, it is essential to apply Large Language Models Using Multi-modal Data for User Modeling in medical data analysis to enhance accuracy, contextual understanding, and interpretability. This research aims to transform the landscape of medical imaging by introducing an interpretable approach for generating precise medical captions from X-ray images and associated data. Current methods in this domain often have some limitations such as insufficient accuracy, lack of contextual understanding, and limited ability to provide explainable results. Addressing these challenges, our work emphasizes the importance of explainable AI (XAI), which is crucial for enhancing trust and reliability in automated medical diagnostics. We propose a novel framework that integrates advanced deep learning techniques with explainable AI. Initially, the framework employs a U-net architecture for pre-processing the medical images to identify the regions of interests (ROIs), effectively isolating and enhancing relevant features. The Vision Transformer (ViT), known for its effectiveness in handling image data, extracts critical features and generates tokens. To further enhance the contextual understanding of the medical images, we construct a comprehensive medical language graph (MLG). This graph is formed by integrating the tokens generated by the ViT with corpus of medical tokens. This integration enables a deeper understanding of the medical context, significantly improving the accuracy and relevance of the generated captions. For the final step, a language decoder is employed to translate the rich, context-aware features into coherent and medically accurate captions. This step is pivotal in transforming complex medical data into understandable and actionable insights. To ensure that our model's decisions are transparent and interpretable, we apply Layer-wise Relevance Propagation (LRP) and Local Interpretable Model-agnostic Explanations (LIME) for the image and text data, respectively. These explainability techniques enable us to provide clear insights into the model's reasoning process, thereby making it more trustworthy and accessible to medical professionals.