Beyond the black box: lessons in explainability from AI in mammography

Abstract With AI already in clinical use, mammography serves as a critical test-bed for the challenges and potential of medical AI. However, its progress is hampered by the ‘black box’ nature of current AI algorithms, limiting clinician trust and transparency. This review analyses the field of Explainable AI (XAI) as a solution, examining its motivations, methods, and metrics. We find the field is dominated by post-hoc saliency methods that provide plausible but not necessarily faithful explanations of AI decision-making. This focus has led to an evaluation gap, where localization accuracy is used as a proxy for explanatory quality without verifying the model’s true reasoning. Inherently interpretable models that could offer more faithful insights are rarely implemented, and a lack of human-centred studies further obscures the clinical utility of current XAI techniques. We argue that for AI in mammography to realize its full potential, the field must urgently shift focus from creating plausible explanations to developing and validating inherently interpretable systems that provide faithful, clinically meaningful insights.