Silent Drift and Site-level Variance in Clinical AI: Real-world performance and subgroup analysis for post-market monitoring of an FDA-cleared artificial intelligence (AI) triage tool for high-risk use case of pulmonary embolism

Background. FDA-cleared artificial intelligence (AI) triage tool for pulmonary embolism (PE) detection has been widely used; however, its performance has not been analyzed for subgroups, clinical sites, and temporal stability. Methods. This retrospective, six-site health system study evaluated the Aidoc BriefCase PE triage model (K190072) across 14,736 CT pulmonary angiography (CTPA) examinations between July 2023 and March 2026. Reference-standard PE characteristic labels were extracted from radiology reports using a zero-shot LLM pipeline validated against a radiologist-annotated 600 reports (κ =0.98). Model performance was evaluated across demographics, embolus characteristics, and site. Temporal behavior was evaluated on operating point by logistic regression, with and without site adjustment, and by segmented regression. Results. PE was present in 1,402/14,736 examinations (9.5%). Overall sensitivity was 64.1% (95% CI, 61.5-66.6%), specificity 98.8% (95% CI, 98.6-99.0%), compared with 93% sensitivity and 93.7% specificity in the FDA 510(k) summary. Sensitivity fell with peripherality (central 88.6%, lobar or segmental 63.1%, subsegmental 33.3%) and with chronicity (acute 65.5%, chronic 57.4%). Sensitivity differed significantly between the two highest-volume sites, 68.7% and 55.7% (P < .001), with CT scanner as the only variable different between the two sites. The false-positive rate fell 40% per year (P<.001) at a similar rate at two sites, while sensitivity fell 15% per year at site A (P<.036), from 72.5% to 65.0% between two halves of the study period. The manufacturer did not report model updates or threshold changes during the study period. Conclusions. Deployed sensitivity was substantially below the cleared benchmark and varied 13 points by site. The gradual shift in high specificity at the cost of low sensitivity, either from data drift or threshold change, either not monitored or not documented by the manufacturer, highlights the importance of independent continuous post-market monitoring.

Authors

Institutions

Publication Details

Journal
medRxiv
Published
2026-11-12
DOI
https://doi.org/10.64898/2026.11.10.26367739
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Silent Drift and Site-level Variance in Clinical AI: Real-world performance and subgroup analysis for post-market monitoring of an FDA-cleared artificial intelligence (AI) triage tool for high-risk use case of pulmonary embolism

Michał Kawzowicz, Callum S. Newton, Palak G. Patel, Kent M. Ogden et al.
medRxiv
Artificial Intelligence in Healthcare and Education
preprint

Silent Drift and Site-level Variance in Clinical AI: Real-world performance and subgroup analysis for post-market monitoring of an FDA-cleared artificial intelligence (AI) triage tool for high-risk use case of pulmonary embolism

Michał Kawzowicz, Callum S. Newton, Palak G. Patel, Kent M. Ogden, Ketan Dayma, Ryan Curl
preprint en

Abstract

Background. FDA-cleared artificial intelligence (AI) triage tool for pulmonary embolism (PE) detection has been widely used; however, its performance has not been analyzed for subgroups, clinical sites, and temporal stability. Methods. This retrospective, six-site health system study evaluated the Aidoc BriefCase PE triage model (K190072) across 14,736 CT pulmonary angiography (CTPA) examinations between July 2023 and March 2026. Reference-standard PE characteristic labels were extracted from radiology reports using a zero-shot LLM pipeline validated against a radiologist-annotated 600 reports (κ =0.98). Model performance was evaluated across demographics, embolus characteristics, and site. Temporal behavior was evaluated on operating point by logistic regression, with and without site adjustment, and by segmented regression. Results. PE was present in 1,402/14,736 examinations (9.5%). Overall sensitivity was 64.1% (95% CI, 61.5-66.6%), specificity 98.8% (95% CI, 98.6-99.0%), compared with 93% sensitivity and 93.7% specificity in the FDA 510(k) summary. Sensitivity fell with peripherality (central 88.6%, lobar or segmental 63.1%, subsegmental 33.3%) and with chronicity (acute 65.5%, chronic 57.4%). Sensitivity differed significantly between the two highest-volume sites, 68.7% and 55.7% (P < .001), with CT scanner as the only variable different between the two sites. The false-positive rate fell 40% per year (P<.001) at a similar rate at two sites, while sensitivity fell 15% per year at site A (P<.036), from 72.5% to 65.0% between two halves of the study period. The manufacturer did not report model updates or threshold changes during the study period. Conclusions. Deployed sensitivity was substantially below the cleared benchmark and varied 13 points by site. The gradual shift in high specificity at the cost of low sensitivity, either from data drift or threshold change, either not monitored or not documented by the manufacturer, highlights the importance of independent continuous post-market monitoring.

medRxiv
SUNY Upstate Medical University (US), Albany Medical Center Hospital (US)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon