Abstract WP270: Comparative Evaluation of Two Commercial AI Solutions for Vessel Occlusion Detection in a Real-World Clinical Setting

Purpose: To compare the performance of two FDA-cleared artificial intelligence (AI) solutions, Vendor A (Aidoc, Tel Aviv, Israel) and Vendor B (Viz.ai, San Francisco, USA) in detecting vessel occlusions (VOs) on CT head angiograms within a large academic healthcare environment. Materials and Methods: This retrospective study included cases processed by both Vendor A and Vendor B over an 18-week period (June 1–September 27, 2024). A natural language processing (NLP) algorithm classified radiology reports. When NLP, vendor A, and vendor B outputs were concordant, this was assumed as ground truth; cases with any discordance underwent review and adjudication to establish ground truth. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Results: A total of 1,557 consecutive CT angiogram cases were reviewed. NLP algorithm identified 113 true positive reports, corresponding to a prevalence of 7.3% (113/1,557). With 402 cases exhibiting discordance. Vendor A demonstrated a sensitivity of 92.4%, specificity of 97.6%, PPV of 76.8%, and NPV of 99.3%. Vendor B demonstrated a sensitivity of 35.4%, specificity of 98.1%, PPV of 62.2%, and NPV of 94.5% as seen in Table 1. Vendor A detected 92.6% of Large Vessel Occlusions (LVO) and 92.3% of Medium Vessel Occlusions (MeVO). Vendor B detected 70.4% of LVOs and only 17.3% of MeVOs, highlighting notable performance differences in MeVO detection as seen in Table 2. Conclusion: In this real-world clinical comparison, Vendor A demonstrated higher sensitivity and stronger agreement with ground truth compared to Vendor B in detection of VO. Vendor A detected 92.6% of LVOs compared to 70% for Vendor B, with the overall performance gap attributable to Vendor A’s greater accuracy in identifying MeVOs.These findings provide valuable comparative data and underscore the importance of evaluating both diagnostic accuracy and statistical agreement when selecting AI solutions for stroke triage workflows. Clinical Relevance Statement: AI algorithms designed for VO detection may vary significantly in diagnostic performance and clinical integration. This study emphasizes the importance of critical vendor assessment, particularly regarding detection sensitivity and agreement with expert adjudication, to guide optimal AI adoption in acute stroke care environments.