Reliability-Driven Label Enhancement in Crowdsourcing: Integrating Confidence Filtering and Collaborative Correction

Crowdsourcing has been widely used for labeling data in artificial intelligence and machine learning due to its cost-saving benefits. However, inconsistencies in worker-provided labels degrade the quality of the integrated labels. Most noise correction methods proposed by researchers in recent years focus on training a corrector using subsets of data that are filtered to be clean. These methods depend on limited data to train the corrector model, which inevitably compromises the corrector’s reliability. To address this problem, this paper aims to enhance the reliability of integrated labels by applying a novel noise correction method that uses correctors trained on both clean and noisy subsets. This method comprises two stages: confidence filtering and an improved co-teaching+ algorithm. In the first stage, this method calculates the confidence levels of the integrated labels for each instance in the crowdsourced dataset, and then divides the dataset into clean and noisy subsets according to the confidence level. In the second stage, our method initially trains a corrector, denoted as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">clean</sub>, on the clean dataset. Subsequently, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">clean</sub> guides the co-teaching+ algorithm to adaptively train two correctors on the complete crowdsourced dataset, which includes both clean and noisy subsets. Finally, the instances in the crowdsourced dataset are corrected by these three correctors. Within this study, comparative experiments have been carried out on fifteen simulated datasets and four real-world datasets. The proposed noise correction method attains the best outcomes on the majority of the datasets evaluated.