AI12 Prospective, multicentre, real-world sensitivities of an artificial intelligence as a medical device assessment and teledermatologist assessment in appropriately managing melanoma, squamous cell carcinoma and rare skin cancers referred on the ur

Abstract Melanoma, squamous cell carcinoma (SCC) and rare cancers are high-risk lesions that should be managed via the National Health Service Urgent Suspected Cancer (USC) pathway. This study reports on the real-world sensitivities of assessments conducted by a UK Conformity Assessed class IIa artificial intelligence as a medical device (AIaMD) and teledermatology review in generating appropriate management outputs for melanoma, SCC and rare cancers. In total, 25 788 lesions referred to the USC pathway were prospectively assessed by both the AIaMD and 81 teledermatologists across 12 National Health Service sites between December 2023 and October 2024. Ground truth outcomes were available for 18 744 lesions (72.7%): histopathology for all cancers, and histopathology or single-clinician labelling for precancerous or benign lesions (noting that single-clinician reads may be associated with a risk of validation bias). All lesions were first assessed by the AIaMD and subsequently by teledermatology. Macroscopic and dermoscopic images, patient and lesion history, and the AIaMD output were available to the teledermatologist. All 264 melanomas (including 109 in situ), 356 SCCs (including 81 keratoacanthomas) and 10 rare cancers were included in the analysis. Management true positives are defined as instances when the recommended next step for a melanoma, SCC or rare cancer was progression on the USC pathway. Point estimate sensitivities are presented with 95% confidence intervals (CIs; Wilson method). Sensitivity comparisons are made using two-tailed two-proportion Z-tests (significance threshold P < 0.05). Overall, the AIaMD was significantly more likely to recommend triaging high-risk skin cancers along the USC pathway (98.6%, 95% CI 97.3–99.2) than teledermatologists (95.9%, 95% CI 94.0–97.2); P = 0.004. Of the nine AIaMD false negatives, eight (89%) were overturned to the USC pathway by teledermatology review. Of the 26 teledermatologist false negatives, 25 (96%) had been correctly routed to the USC pathway by the AIaMD. The AIaMD was significantly more likely to recommend triaging melanomas along the USC pathway (98.1%, 95% CI 95.7–99.2) than teledermatologists (94.7%, 95% CI 91.3–96.8); P = 0.04. The AIaMD was significantly more likely to recommend triaging SCCs along the USC pathway (99.2%, 95% CI 97.5–99.7) than teledermatologists (96.6%, 95% CI 94.2–98.1); P = 0.02. In conclusion, these data suggest that, for the task of triaging high-risk skin cancers to the USC pathway, outputs from the AIaMD are at least as good as teledermatology review. This study adds further evidence to benchmarking the sensitivity performance target of the AIaMD at > 95% for melanoma and SCC. All authors are employed by or affiliated with the AIaMD manufacturer.