Abstract 4146391: Multinational Validation of Fully Automated Diagnostic Reports using Artificial Intelligence-enabled Application for ECG Images

Background: Accurate ECG interpretation is critical for triaging, diagnosis, and managing patients with cardiovascular conditions. Current computerized methods have limited accuracy and are often proprietary algorithms based on raw signal data. We sought to evaluate ECG-GPT, a novel vision-text transformer model capable of generating free-text diagnosis statements from images of ECGs. Methods: ECG-GPT was developed with 3 million ECGs at Yale and validated in 2 large and geographically distinct populations: (1) 1.4 million ECG images with diagnosis statements from Mount Sinai Health System (MSHS), NY, and (2) 45,389 ECG images with 6 diagnostic labels from UK Biobank (UKB). We employed 2 metrics: (A) a rule-based approach to evaluate diagnostic accuracy for 19 rhythm and conduction disorders on MSHS ECGs and 6 key conditions on UKB ECGs, and (B) semantic similarity (similar meaning) using a fine-tuned DistilBERT model. Using semantic similarity methods, we assessed the quality of generated text across the full range of ECG diagnoses by computing similarity both for all model-generated and reference statements and for subsets specific to ECGs flagged for each of the 19 extracted conditions. To establish baseline similarities, we also computed semantic similarities between random combinations of model-generated and reference statements within each of these sets. Results: The model performed well across multiple key labels, spanning rhythm and conduction disorders, including an AUROC of 0.92 for atrial fibrillation, 0.95 for left bundle branch block, and 0.90 for atrioventricular block (Table A). More importantly, in addition to the labels, ECG-GPT identified the full context of the diagnosis statements with allied conditions. It had a median pairwise cosine similarity of 0.86 (IQR 0.78-0.94), significantly greater than the median baseline similarity of 0.73 (IQR 0.66-0.79, p<0.001) (Table B). This separation between median pairwise and baseline similarity remained consistent across all 19 condition-specific subsets. Results were comparable in the UKB dataset, with AUROCs of 0.95, 0.92, and 0.95 for AF, ST, and SB, and AUROCs of 0.99, 0.99, and 0.95 for LBBB, RBBB, and AVb (Table C). Conclusion: Our model represents a scalable and accessible strategy for generating accurate, expert-level reports from photos of ECGs, validated across large, multinational datasets using both individual diagnostic labels and the comprehensiveness of the diagnostic statement.