Motivation Matters: Challenges and Pit-Falls of Crowdsourced Online Studies

Conducting online studies via crowdsourcing platforms is a widely used approach in the field of pervasive computing, as it enables efficient access to diverse, and scalable participant samples. However, certain study designs are not well suited for online implementation, risking compromised data quality. This issue became evident during our study on human interaction with AI assistance. By leveraging Prolific, a crowdsourcing website enabling online studies, we investigated how humans interact with AI-generated hints while solving a pipe maze game. The study employed "Poor Man’s Eye Tracking," combining mouse tracking with obscured vision fields, to monitor participant behaviour. Contrary to our hypothesis, participants did not increase diligence in verifying AI hints after AI errors were pointed out. Analysing our data suggests a lack of intrinsic motivation and attention among participants. This falsified the reliability and validity of the collected data. Our study showed that even a careful pre-selection of participants through a crowdsourced website cannot prevent these issues. Moreover, identifying missing motivation or inattentiveness in non-questionnaire components, such as game scenarios, requires additional data collection and analysis, such as mouse tracking, to retrospectively filter invalid datasets. This paper discusses potential causes and methods to mitigate these issues and proposes strategies to identify and prevent data distortion caused by missing motivation and inattentiveness of subjects.