From One Case to a Pattern: Convergence-Based Label Noise Detection Across Noise Types and Class Sizes

Reliable evaluation of machine learning systems depends on trustworthy labels, but existing label-noise detectors typically require either ground truth to validate against or expensive per-example computation to identify errors. We show that a simpler, aggregate signal is already available: when several class-balancing techniques are applied to the same dataset, their disagreement on target-class recall reliably tracks the amount of label noise present, offering evaluators a way to flag potentially unreliable training or evaluation labels without needing ground truth. We validate this signal rigorously, not just observationally: across random and systematic noise injection, three target-class sizes, and two model architectures, using the coefficient of variation (CV) as a metric corrected for two confounds identified during initial testing (a floor effect and outlier domination from one resampling method), and confirmed statistically through repeated-seed experiments (5 to 10 independent seeds per condition) with 95% confidence intervals throughout. The core trend holds robustly in five of six model-by-class-size combinations tested; the sixth, a neural network on a middle-sized class, reveals a genuine architecture-specific limit to the signal's reliability rather than a uniform guarantee. We further compare this aggregate signal against confident learning, an established per-example detector: both track injected noise reliably, but CV is substantially more computationally expensive, a structural consequence of requiring multiple model fits to measure disagreement rather than a single calibrated classifier. We identify a concrete technical setting where an aggregate signal is nonetheless preferable, classifiers without calibrated probability outputs, and report these findings, their architecture-dependent behavior, and open questions for developing this signal into a practical diagnostic tool as directions for future work.