Challenges and Opportunities for Survey Research in the Age of Generative AI: An Experience Report

Survey research, while common, has known challenges and limitations. In this paper, we discuss the challenges and concerns of conducting research using online survey in the age of widespread development and use of generative AI technologies. We base our discussion on our recent experiences conducting survey research to better understand software practitioners’ knowledge of and experience with automated program repair (APR). In our efforts, we encountered both anticipated and unexpected challenges, much of which stemmed from advancements in AI-assisted automation (e.g., generative AI). For example, we found that many of the open ended responses provided were likely generated by AI. Based on our experiences, we outline how we mitigated or handled these challenges and discuss opportunities for improving survey design and administration to ensure high quality research and outcomes in the age of advanced and widely available AI technologies.