Improving Uncertainty Estimation with Confidence-Aware Training Data

AI-driven second-opinion systems play a crucial role in decision-making, especially in medicine, where accurate predictions guide clinicians. However, quantifying uncertainty in deep learning is challenging, as current methods often rely on hard class labels, which do not reflect true prediction confidence. This often results in overconfident predictions and slow convergence to true probabilities. To address this, we suggest a new method that separates uncertainty into two types: epistemic and aleatoric. We estimate these uncertainties using hard and soft confidence labels, with experts providing confidence levels that indicate the likelihood of misclassification. We release an updated blood typing dataset consisting of 3139 images with soft labels of uncertainty annotations from six experts and hard labels collected from medical records. Proposed approach improves SotA uncertainty estimation quality by two times for blood typing (classification) and by 62% for histology (segmentation)<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>The code is available at: https://github.com/createcolor/confidence-aware-uncertainty..