ConsensusXAI: A framework to examine class-wise agreement in medical imaging

Explainable AI (XAI) is essential for trust and transparency in deep learning, especially in medical imaging.<br/>Existing local explanation methods provide per-instance insights but fail to show whether similar explanations<br/>hold across samples of the same class. This limits global interpretability and demands time-consuming manual<br/>review by clinicians to trust models in practice. We introduce the Consensus Alignment Score (CAS), a<br/>novel metric that quantifies consistency of explanations at the class level. We also present ConsensusXAI, an<br/>open-source, model- and method-agnostic framework that evaluates explanation agreement quantitatively (via<br/>CAS) and qualitatively (through consensus heatmaps) per class. Unlike prior benchmarks, ConsensusXAI uses<br/>a latent-space clustering approach, Latent Consensus, to identify dominant explanation patterns, exposing<br/>biases and inconsistencies towards certain classes. Evaluated across two different medical imaging modalities for<br/>both correct and incorrect predictions on two different backbones, our method consistently reveals meaningful<br/>class-level insights, outperforming traditional consensus method i.e. SSIM, and enabling faster, more confident<br/>clinical adoption of AI models.