Evaluating four consumer artificial intelligence chatbots on smartphone dermatology images: diagnostic accuracy, the contribution of clinical context, and skin-tone equity

Abstract Patients increasingly upload smartphone photographs to consumer artificial intelligence (AI) chatbots seeking dermatological diagnoses, yet accuracy on such images is unexplored. We compared the diagnostic accuracy and equity of four leading AI chatbots across varying input conditions. GPT-5.4-Pro, Claude-Opus-4.6, Gemini-3.1-Pro and Grok-4.20-Reasoning were evaluated on 3,005 cases from the crowdsourced Skin Condition Image Network (SCIN) dataset with physician-validated ground truth. Each case was assessed under four conditions: single image (n = 1,073), multiple images (n = 1,932), images plus clinical context (n = 2,567) and text-only (n = 2,567), using top-1 and top-3 hierarchical matching. In the primary analysis (images plus context), models clustered into two tiers: Grok (53.2%, 95% CI 51.3–55.1) and Gemini (52.6%, 95% CI 50.7–54.5) significantly outperformed GPT (45.5%, 95% CI 43.5–47.5) and Claude (44.6%, 95% CI 42.7–46.5) for top-1 accuracy (Cochran’s Q, p < 0.001). Gemini led top-3 accuracy at 79.3% (95% CI 77.7–80.8). Adding clinical context improved accuracy for Grok (+ 5.5%, p < 0.001) and GPT (+ 3.8%, p = 0.003) only, whereas removing images caused losses of 11 to 18 percentage points across all models. Inter-model agreement was low across image-based analyses (Fleiss' kappa 0.34–0.41). For top-1 accuracy, Fitzpatrick skin tone disparity was significant only for Grok, the best-performing model (56.0% light versus 45.1% dark skin, p = 0.007), whereas top-3 disparities were significant for Claude, Gemini and Grok. Consumer AI chatbots show promise, but skin tone disparities, low inter-model agreement and content moderation limitations indicate these tools are not ready for standalone clinical use.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-10683647/v1
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Evaluating four consumer artificial intelligence chatbots on smartphone dermatology images: diagnostic accuracy, the contribution of clinical context, and skin-tone equity

Fatmah Alzahrani, Matthew R. Hall, Kevin Tabatabaei
Research Square
Artificial Intelligence in Healthcare and Education
preprint

Evaluating four consumer artificial intelligence chatbots on smartphone dermatology images: diagnostic accuracy, the contribution of clinical context, and skin-tone equity

Fatmah Alzahrani, Matthew R. Hall, Kevin Tabatabaei
preprint en

Abstract

Abstract Patients increasingly upload smartphone photographs to consumer artificial intelligence (AI) chatbots seeking dermatological diagnoses, yet accuracy on such images is unexplored. We compared the diagnostic accuracy and equity of four leading AI chatbots across varying input conditions. GPT-5.4-Pro, Claude-Opus-4.6, Gemini-3.1-Pro and Grok-4.20-Reasoning were evaluated on 3,005 cases from the crowdsourced Skin Condition Image Network (SCIN) dataset with physician-validated ground truth. Each case was assessed under four conditions: single image (n = 1,073), multiple images (n = 1,932), images plus clinical context (n = 2,567) and text-only (n = 2,567), using top-1 and top-3 hierarchical matching. In the primary analysis (images plus context), models clustered into two tiers: Grok (53.2%, 95% CI 51.3–55.1) and Gemini (52.6%, 95% CI 50.7–54.5) significantly outperformed GPT (45.5%, 95% CI 43.5–47.5) and Claude (44.6%, 95% CI 42.7–46.5) for top-1 accuracy (Cochran’s Q, p < 0.001). Gemini led top-3 accuracy at 79.3% (95% CI 77.7–80.8). Adding clinical context improved accuracy for Grok (+ 5.5%, p < 0.001) and GPT (+ 3.8%, p = 0.003) only, whereas removing images caused losses of 11 to 18 percentage points across all models. Inter-model agreement was low across image-based analyses (Fleiss' kappa 0.34–0.41). For top-1 accuracy, Fitzpatrick skin tone disparity was significant only for Grok, the best-performing model (56.0% light versus 45.1% dark skin, p = 0.007), whereas top-3 disparities were significant for Claude, Gemini and Grok. Consumer AI chatbots show promise, but skin tone disparities, low inter-model agreement and content moderation limitations indicate these tools are not ready for standalone clinical use.

Research Square
University of British Columbia (CA), Mayo Clinic in Florida (US)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon