Refining Deep Active Learning Pipelines: Leveraging Pseudo-Labeling and Label Calibration for Enhancing Dataset Quality

The performance of Deep Learning (DL) models largely depends on the quality and quantity of labeled data used for training. In recent years, the demand for high-quality, large-scale datasets has significantly increased as AI models have grown in size and complexity. However, human annotation is costly, making it challenging to gather sufficient labeled data. Active Learning (AL), and particularly Deep Active Learning (DeepAL) has emerged as a promising approach for efficiently improving datasets. DeepAL aims to maximize the learning efficiency with minimal annotation effort by selecting data that provides critical information for model training. In this paper, we propose an effective and efficient process for improving datasets by incorporating pseudo-labeling and label calibration methods into DeepAL. This approach was evaluated on a multi-label classification task on a custom image dataset. By using Monte Carlo Dropout (MC-Dropout) for robust inference on unlabeled data, we achieve the following three objectives simultaneously. (1) Mislabeling calibration (2) Selection of highly reliable pseudolabels (3) Sampling of annotated data that is both diverse and informative In our experiments, we synthesized a noisy dataset by intentionally mislabeling 50% of the samples. This work presents a new pipeline that integrates and improves upon existing DeepAL methods into a unified framework. We are progressively improving the dataset quality, assessing each stage by fine-tuning a model on the updated dataset and measuring its performance. The results confirmed the effectiveness of the proposed process, leading to step-by-step improvements in dataset quality.