Can digital humanities use microwork crowdsourcing in a fair manner? The effect of pedagogical training and multimodal instructions on annotation quality

Abstract This article explores the potential of microwork crowdsourcing for digital humanities. In contrast to volunteer-based crowdsourcing, microwork involves paying non-expert workers recruited through online platforms for their work. Microwork crowdsourcing is already an established part of the academic research infrastructure in many data-driven fields. However, previous research has identified numerous issues regarding worker treatment and experience, which range from sweatshop wages to unpaid labor. Building on previous research in human–computer interaction, this study evaluates two mechanisms designed to promote fairness: multimodal instructions, which combine language and visual content, and pedagogical training, which provides multiple examples and feedback. We tested their impact on data quality through two experiments, in which crowdsourced workers perform complex tasks that involve reasoning about primary school science diagrams. While pedagogical training did not lead to significant improvements, multimodal instructions had a positive effect on data quality. Our results suggest that microwork crowdsourcing has potential for digital humanities, especially in fields unlikely to attract volunteer communities. However, these findings highlight the importance of fairness-oriented task design and transparency in the responsible use of crowdsourcing.