Parse Tree Tracking Through Time for Programming Process Analysis at Scale

Background and Context: Programming process data can be utilized to understand the processes students use to write computer programming assignments. Keystroke- and line-level event logs have been used in the past in various ways, primarily in high-level descriptive statistics (e.g., timings, character deletion rate, etc). Analysis of behavior in context (e.g., how much time students spend working on loops) has been cumbersome because of our inability to automatically track high-level code representations, such as abstract syntax trees, through time and unparseable states. Objective: Our study has two goals. The first is to design the first algorithm that tracks parse tree nodes through time. Second, we utilize this algorithm to perform a partial replication study of prior work that used manual tracking of code representations, as well as other novel analyses of student programming behavior that can now be done at scale. Method: We use two algorithms presented in this paper to track parse tree nodes through time and construct tree representations for unparseable code states. We apply these algorithms to a public keystroke data from student coursework in a 2021 CS1 course and conduct analysis on the resulting parse trees. Findings: We discover newly observable statistics at scale, including that code is deleted at similar rates inside and outside of conditionals and loops, a third of commented out code is eventually restored, and that frequency with which students jump around in their code may not be indicative of struggle. Implications: The ability to track parse trees through time opens the door to understanding new dimensions of student programming, such as best practices of structural development of code over time, quantitative measurement of what syntactic constructs students struggle most with, refactoring behavior, and attention shifting within the code.