An offline smartphone skin-triage model that failed external validation: a negative-result report

Abstract Smartphone tools that place skin lesions into triage bands are often proposed for areas with too few dermatologists. Yet published studies tend to focus on internal performance, while external validation remains uncommon, particularly for pigmented skin. I evaluated a frozen, fully offline Android triage model (dermedge-1.8.0; MobileNetV2, INT8-quantized TensorFlow Lite, an eight-class classifier trained on HAM10000, PAD-UFES-20, and Fitzpatrick17k) using a scoring-only protocol on three external datasets not used for training: DDI, DDI-2, and MRA-MIDAS. The app converts the model's malignancy score into triage bands, including REFER. An uncertainty gate returns "cannot assess" rather than displaying a band when output is uncertain. In total, 3,266 images were scored, of which 3,034 had usable labels. On the model's own held-out test split, lesion-only AUC was 0.948. External in-scope AUCs were 0.683 [0.624, 0.740] for DDI, 0.719 [0.632, 0.805] for DDI-2, and 0.692 [0.657, 0.727] for MRA-MIDAS. Since the intervals overlap, the results indicate similar performance, around 0.68 to 0.72, rather than a ranking across datasets. For in-scope DDI images labelled Fitzpatrick V–VI (81 images, 16 malignant), AUC was 0.588 [0.404, 0.753], indistinguishable from chance. The app returned "cannot assess" for 72–80% of external images and displayed REFER for 7–18%. Among 1,057 malignant images, 808 received "cannot assess" and 41 appeared in the reassuring LOW band. The model failed external validation by a wide margin and is not ready for a pilot. I report these results in full because the internal–external gap, dark-skin performance, and abstention burden are central to any responsible deployment decision. No clinical pilot should proceed until a frozen model meets a pre-registered external-performance bar.