Do comments and expertise still matter? An experiment on programmers’ adoption of AI-generated JavaScript code

This paper investigates the factors influencing programmers’ adoption of AI-generated JavaScript code recommendations within the context of lightweight, function-level programming tasks. It extends prior research by (1) utilizing objective (as opposed to the typically self-reported) measurements for programmers’ adoption of AI-generated code and (2) examining whether AI-generated comments added to code recommendations and development expertise drive AI-generated code adoption. We tested these potential drivers in an online experiment with 173 programmers. Participants were asked to answer some questions to demonstrate their level of development expertise. Then, they were asked to solve a LeetCode problem without AI support. After attempting to solve the problem on their own, they received an AI-generated solution to assist them in refining their solutions. The solutions provided were manipulated to include or exclude AI-generated comments (a between-subjects factor). Programmers’ adoption of AI-generated code was gauged by code similarity between AI-generated solutions and participants’ submitted solutions, providing a behavioral measurement of code adoption behaviors. Our findings revealed that, within the context of function-level programming tasks, the presence of comments significantly influences programmers’ adoption of AI-generated code regardless of the participants’ development expertise. • Comments in AI-generated code significantly boost adoption among programmers, re- gardless of development expertise. Both novice and expert programmers were more likely to integrate AI-generated code when comments were present. • Development expertise does not significantly affect the adoption of AI-generated code, suggesting that supportive code features matter more than expertise alone. • We developed a novel objective measure of AI code adoption by applying PCA to four code similarity metrics. This combined metric captures how much AI-generated code participants incorporated, offering a robust way to quantify adoption behavior in programming studies.