Identifying and mitigating the influence of invalid responses in online surveys: a longitudinal case study

Abstract Use of online surveys is common in educational research. It is well known that artificial intelligence (AI) bots are highly active throughout the internet. Yet, the full impact of bots and their influence in online educational research is not fully understood. The following report describes the iterative methodology and results for identifying and removing suspected bot responses used by one research team across three online, national surveys in the United States and how these processes changed over time. Of the 2,116 responses received across the three surveys, 1,596 responses (75.4%) were removed. Of the 1,596 responses removed, 1,070 responses (67.0% of removed responses) were confidently identified as bots by the research team. Strategies for identifying obvious and non-obvious bot responses in a survey and preventing bots through intentional survey design features are discussed. Implications include improving the reliability of survey data analysis and practices for reporting survey data.