Cost-Optimized Crowdsourcing for NLP via Worker Selection and Data Augmentation

This paper presents worker selection and data augmentation algorithms aimed at improving annotation quality and reducing costs in crowdsourcing for Natural Language Processing (NLP). Unlike previous studies targeting simpler tasks like binary classification, which require less contextual understanding, this study aims to provide a unified paradigm for a wider spectrum of NLP tasks, with sequence labeling and text generation as application showcases. Utilizing a Combinatorial Multi-Armed Bandit (CMAB) approach and a cost-effective human feedback mechanism, the proposed worker selection algorithm effectively addresses the challenge of label inter-dependency in NLP tasks. Additionally, our algorithm tackles the issues presented by imbalanced and small-scale datasets through data augmentation methods. Experiments on the CoNLL 2003 NER, Chinese OEI, and YACLC datasets demonstrated the algorithm's efficiency, achieving up to 100.04% of the expert-only baseline <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\textrm {F}$</tex-math></inline-formula>-score and 65.97% cost savings. A dataset-independent experiment yielded 97.56% of the expert baseline <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\textrm {F}$</tex-math></inline-formula>-score and 59.88% cost savings. We also provide a theoretical analysis proving our worker selection framework achieves sub-linear regret.