Improving Reproducibility in Crowdsourced Labeling Across Platforms

Reproducibility of crowdsourced labeling results across platforms remains a challenge, as differences in worker ability distributions often lead to inconsistent accuracy. This paper addresses this issue by proposing an ability-based approach to improve reproducibility. We hypothesize that matching worker ability distributions across platforms enhances result consistency. Using Latent Rank theory, we model worker abilities from original experiments and create a benchmark test to select participants for replication studies. Our findings show that using worker sets with similar ability distributions significantly improves reproducibility. However, requiring all workers to answer all tasks is impractical. To address this, we develop a collaborative filtering method that reduces the number of required workers to 3–5 per task, enabling reproducibility without additional cost. This approach enables more reliable crowdsourcing for scientific research and AI development.