A Scoping Review of Interpretable AI Model Approaches in Echocardiography: Frameworks, Model Types, Explanation Strategies, and Evaluation Practices

Echocardiography is a central modality for cardiac diagnosis; rising clinical demand and advances in machine learning have accelerated AI development for automated analysis. As these systems may influence high-stakes decisions, model interpretability is essential for clinical trust and accountability. In this review, we define interpretable AI model approaches as strategies that improve interpretability through intrinsically interpretable components within transparent models or post hoc explainable AI (XAI) for black-box models. We synthesized 170 echocardiography studies using interpretable AI model approaches and mapped model types, explanation strategies, and evaluation practices. To compare approach-specific patterns, studies were categorized into multistep and end-to-end frameworks, representing intermediate representation versus direct prediction. Multistep studies were more prevalent and typically combined convolutional neural network (CNN)-based upstream models with intrinsically interpretable downstream approaches. End-to-end studies included more complex architectures, with the most common pairing being CNN-based models with class activation mapping-based visualizations. Evaluation practices also differed by framework. Multistep studies more frequently reported quantitatively defined clinical metrics and included several studies demonstrating prospective, application-level implementation. In contrast, at the model-approach instance level, end-to-end instances relied more often on qualitative-grounded evaluation than multistep upstream instances (23/57, 40.4% vs. 2/122, 1.6%). Collectively, these findings highlight the distinct challenges and promising research directions for each approach: multistep models may benefit from more advanced architectures, whereas end-to-end models require more deliberate post hoc explanation methods and stronger quantitative evaluation to support clinical translation. This work, therefore, provides a structured foundation and classification approach to meaningfully evaluate and advance interpretable AI model approaches in echocardiography.