Confident Labels: A Novel Approach to New Class Labeling and Evaluation on Highly Imbalanced Data

A common challenge in machine learning is obtaining readily available labeled data, as the majority of data remains unlabeled and thereby necessitates human annotation from domain experts, e.g. doctors. Thus, substantial cost is associated with labeling the data, e.g. healthcare diagnostics. The efficient and effective acquisition of labeled data is crucial for early detection, specifically, in relation to cognitive decline. In our work, we employ a novel unsupervised method to generate binary class labels on a publicly available, highly imbalanced cognition dataset derived from the Health and Retirement Study (HRS). To evaluate the efficacy of our newly generated class labels, we also employ a novel approach to evaluate the labels by directly comparing them to ground-truth labels, rather than the traditional approach which measures the performance of a supervised model trained on the generated labels. Our results demonstrate that the newly generated class labels significantly outperform the baseline method, IF, in terms of Balanced Accuracy (BA), Geometric Mean (GM), F-Measure (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$F$</tex> <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf>), and Matthew's Correlation Coefficient (MCC).