Model-dependent accuracy inflation in machine learning evaluation on public sickle cell imaging datasets

Abstract Background Automated sickle cell detection is reported at high accuracy on a small number of shared public image collections, yet whether those collections support the accuracies reported on them has not been measured. This study quantifies how much detection performance is attributable to how the datasets were built rather than to the erythrocytes. Methods Four public corpora were analysed. Repeated images were recovered by dihedral, shift-tolerant correlation verified at two resolutions, with the grouping threshold anchored to a blinded two-rater reference study of 60 image pairs. Inflation was the loss in area under the curve when sources were held out, using hand-crafted features with two learners and frozen and fine-tuned con-volutional networks. File traces were tested with an accuracy-based and a balanced confounding index against permutation nulls; drawn annotations were located by differencing the two released versions of one corpus; and bootstrap optimism correction was compared with the source-held es-timate. Results Similarity separated true repeats from distinct fields with an AUC of 0.978; above 0.55 ev-ery consensus pair was a repeat, whereas a threshold calibrated on synthetic copies recovered only 66% of genuine repeats. With hand-crafted features, holding sources out reduced AUC by 0.094 on a release with twelve duplicated control fields, by 0.025 to 0.084 on a smartphone corpus pho-tographing each field about twice, and by 0.092 under participant-level holdout. A fine-tuned net-work lost 0.021 on the smartphone corpus, its source-held error 1.8 times the naive error, but noth-ing on the duplicated release. The accuracy-based index read zero for traces marking a third of the majority class, which the balanced index recovered. A resolution trace was visible to features and network yet unused, whereas bounding boxes in one corpus’s labelled release raised AUC by 0.058. Bootstrap optimism correction overstated source-held performance, and one field was filed under both classes. Conclusions Inflation depends on the model and must be measured for each model reported. Par-ticipant-level dependence cannot be verified by image-level source recovery, and most releases dis-tribute no identifiers. Six checks, each calibrated to the corpus itself, are proposed as an acceptance test for imaging tools entering screening programmes.