Jiving with LLMs: Assessing First Year Students’ Computer Programming Self Efficacy After Reading Code With LLMs

This study investigated the impact of leveraging generative artificial intelligence (GenAI) to assist 1st-year engineering and computer science (CS) students in reading code in a new (to them) language. Students were asked to comment code in FORTRAN. They were then asked to run the code through ChatGPT-4.0 for its comments and reflect on what they learned from the experience. Participants completed survey items from Ramalingam and Wiedenbeck’s Computer Programming Self-Efficacy Scale (CPSES) prior to and after the intervention. Additional open-ended reflective (qualitative) questions were added to the quantitative questions in the postintervention questionnaire. This study documents increases in self-efficacy for programming independence and persistence (Factor 1 of the CPSES) as well as for complex programming tasks (Factor 2) after students used ChatGPT for generating code explanations. Both CS and engineering students showed improvements in programming independence and persistence; but only engineers showed significant improvements in their confidence regarding complex programming tasks. Men experienced a significant increase in self-efficacy on Factor 1 while women experienced a significant increase on Factor 2. The qualitative data point to an increase in student understanding of the new code and suggest that although students may be more likely to use GenAI for assistance as they progress through programming courses, guiding students in using GenAI to understand code may shift students’ intent away from using GenAI to write code for them. Thus, we recommend that programming faculty instruct students how to interact with GenAI.