Attack of the bots: Lessons from a compromised online MSM survey

Anonymous online surveys using financial incentives are an essential tool for understanding sexual networks and risk factors including attitudes, sexual behaviors, and practices. However, these surveys are vulnerable to bots attempting to exploit the incentive. We deployed an in-person, limited audience survey via QR code at select locations in North Carolina to assess geolocation application use among men who have sex with men to characterize the role of app usage on infection risk and behavior. The survey was unexpectedly posted on a social media platform and went viral. Descriptive statistics were performed on repeat responses, free-text length, and demographic consistency. Between August 2022 and March 2023, we received 4,709 responses. Only 13 responses were recorded over a 6-month period until a sharp spike occurred: over 500 responses were recorded in a single hour and over 2,000 in a single day. Although free-text responses were often remarkably sophisticated, many multiple-choice responses were internally inconsistent. To protect data quality, all online surveys must incorporate defensive techniques such as response time validation, logic checks, and IP screening. With the rise of large language models, bot attacks with sophisticated responses to open-ended questions pose a growing threat to the integrity of research studies.