Examining explainable artificial intelligence in TNM staging with PET-CT: a user-centred observation study

PURPOSE: Artificial intelligence (AI) is increasingly proposed as a solution to improve efficiency in radiology and nuclear medicine, particularly in the context of workforce shortages. However, adoption of AI-based clinical decision support systems (AI-CDSS) remains slow, due to limited model transparency. Explainable AI (XAI) may improve clinician acceptance by supporting oversight and trust. This study evaluated the impact of different XAI explanation types on radiologists' willingness to adopt AI systems. METHODS: Ten nuclear medicine radiologists from eight UK institutions performed lung cancer TNM staging using whole-body PET/CT scans supported by a simulated AI-CDSS. Three explanation approaches were assessed: input feature attribution, high-level concept explanations and global algorithmic transparency. Adoption likelihood and explanation usefulness were rated using Likert scales and analysed with nonparametric sign tests. Semi-structured interviews were additionally analysed through thematic evaluation supported by large language model-assisted coding with human verification. RESULTS: All explanation approaches significantly increased radiologists' willingness to adopt the AI system compared to a black-box model (p < 0.05). Explanations were consistently considered useful in enabling participants to confirm or challenge AI staging recommendations (p < 0.001). Qualitative findings highlighted the importance of clinical relevance, error detection and decision support value. A trade-off between explanation depth and usability was identified as a key factor influencing preferences. CONCLUSION: Incorporating XAI into nuclear medicine CDSS enhances radiologists' acceptance and provides clinically meaningful information for oversight of AI recommendations, in accordance with the EU AI Act. These findings support the role of XAI in facilitating integration of AI tools into diagnostic workflows.