AI16 Performance of a deep neural network in diagnosing lesions referred on the suspected skin cancer pathway: a prospective observational study

Abstract Several countries, including the UK, have approved commercial artificial intelligence (AI) products as medical devices to assist in the diagnosis of skin cancer. However, lack of robust real-world data has led to professional bodies recommending against their use outside research environments. The aim of this study was to assess the performance of an internationally proven deep neural network as a screening tool in patients referred from primary care on the suspected skin cancer pathway. Dermoscopic images were obtained from patients attending a community lesion imaging clinic following referral from primary care on the suspected skin cancer pathway. Ultimate diagnosis was based on histology where available, face-to-face clinical opinion if no histology was obtained, and the opinion of two dermatology specialists with over 10 years’ experience for patients discharged directly from teletriage without histology or face-to-face review. Images were assessed by the neural network and binary categorization was performed. The algorithm was set to detect invasive malignancies and premalignant conditions (actinic keratosis, Bowen disease, keratoacanthoma) and was trained to dismiss benign naevi, benign keratotic lesions (seborrhoeic keratosis, lichen planus-like keratosis, solar lentigines), dermatofibromata and benign vascular lesions. In total, 333 of 382 lesions from consecutive patients who gave research consent were included. Reasons for exclusion were pen artefact on dermoscopic image (26), lesion resolved prior to imaging clinic (eight) inappropriate site (six), poor image quality (five) and other (four). Ultimate diagnoses were basal cell carcinoma (62), squamous cell carcinoma (19), melanoma (11), other malignancy (one), benign keratotic (80), actinic keratosis (61), Bowen disease (20), benign naevus (17), benign vascular (nine), keratoacanthoma (eight) and other benign (45). Overall, 81 of 333 lesions (24.3%) were classified by the algorithm as benign and 252 (75.7%) were flagged as possibly malignant or premalignant. Sensitivity for cancer was 100% (93 of 93 detected). Sensitivity for premalignant conditions was 97% (86 of 89 detected). Specificity for all patients was 51.7% (78 of 151 benign lesions correctly classified). Specificity when removing the ‘other benign’ category, which is in the algorithm’s exclusion criteria and not part of its training, was 65.1% (71 of 109 lesions accurately categorized). Two actinic keratoses and one cases of Bowen disease were incorrectly classified as benign. The algorithm performed extremely well on this real-life case series and detected 100% of invasive malignancies. AI screening of suspected skin cancer referrals is likely to play a significant part in addressing future National Health Service capacity–demand mismatch and would have reduced the caseload requiring specialist opinion by 24.3% in this series. There is urgent need in the UK for further transparent clinical trials and open competition between AI products on large real-world datasets. Further stratification of benign and premalignant conditions may lead to greater benefit than binary classification.