From visual saliency to anatomical explanation: generating textual diagnostic reports from brain MRIs with saliency-grounded LLMs

The opaque nature of deep learning models remains a significant barrier to their clinical adoption in medical imaging. This paper presents a systematic empirical benchmark of visual saliency attribution, atlas-based visualisation, and LLM-based reporting for brain tumour MRI classification, integrated into a unified pipeline, leveraging large language models (LLMs) to deliver human-interpretable diagnostic narratives. The proposed framework operates through three coupled stages. First, nine CNN architectures are extended with a dual-output hybrid formulation that simultaneously optimises a classification head and a segmentation head, enabling spatially richer feature learning. Second, visual saliency attribution methods, namely Grad-CAM, Grad-CAM++, and ScoreCAM, are applied to generate class-discriminative heatmaps, which are subsequently refined into coarse binary masks via an adaptive percentile thresholding pipeline. Third, the resulting masks are projected onto the Harvard–Oxford cortical atlas as an illustrative anatomical overlay, offering an approximate visual reference for the tumour’s cortical neighbourhood, and the extracted findings are encoded into a structured JSON file that conditions three LLMs (Grok3, Mistral, and LLaMA) to generate coherent, radiological-style diagnostic reports. Evaluated on a dataset of 4,834 contrast-enhanced T1-weighted brain MRI images spanning three tumour classes, InceptionResNetV2 achieved the highest classification performance, and Grad-CAM++ yielded the best segmentation overlap. Among the language models, Grok3 led in lexical diversity and coherence, while LLaMA achieved the highest readability score. By integrating visual, anatomical, and linguistic modalities into a unified pipeline, the framework attempts to produce technically grounded and meaningfully interpretable explanations, offering a step toward more transparent artificial intelligence-assisted brain tumour diagnosis.