Identifying Non-Cooperative Participation in Web-Based Elicitation of Acceptability Judgments: How to Get Rid of Noise in Your Data

In this paper, we discuss different sources of noise or other detrimental effects in the elicitation of experimental data. These effects may emerge due to the loss of control in unsupervised web-based elicitation, be task-related, or emerge due to questionnaire designs that lack sufficient means to identify inapt participants. We describe a stepwise process to reach elicited data at the highest attainable level in web-based Acceptability Judgment Tasks (AJTs). In the first step, the questionnaire design, we focus especially on a careful construction of appropriate filler and control items and introduce an alternative to instructional manipulation checks appropriate for AJTs, namely attention items. The second step is to choose the right platform to elicit experimental data from. Lastly, we will show how to employ latency- and response-based methods – analyses of response times as well as of responses to the specialized items – to reliably detect inapt participants.