Understanding Comparative Comprehension Barriers for Students during Code Review through Simplification

Comparative code comprehension - the cognitive activity of understanding how two pieces of code behave relative to each other - occurs during code review, code search, and debugging. Recent research revealed barriers that prevent students from effectively practicing comparative code comprehension during code review and determining whether a code change is behavior-preserving. This prior work suggests it is a difficult task for students, with an accuracy of 44%. However, the tasks in prior work had high complexity, potentially leading to artificially low success rates for students. In this work, we also study comparative code comprehension with students but with simplified tasks in an effort to reduce the previously identified barriers and set students up for high levels of success. We conjecture that simplification ensures that the failures and barriers students encounter are as foundational as possible. Specifically, we target barriers within the tool, code, and comparative comprehension categories from prior work. We conduct an empirical study with 125 students and eight tasks of varying sizes with code changes of varying complexity. We discover that with the chosen experimental setup, students could successfully recognize changes in code behavior with an accuracy of 89%. When students made mistakes, these fell into two primary categories: misunderstanding of behavior preservation and shallow or incomplete evaluation of change logic. Our results can benefit software engineering educators seeking to incorporate code review into the classroom while minimizing barriers.