How do copilot suggestions impact developers’ frustration and productivity?

Abstract Context AI-based development tools, such as GitHub Copilot, are transforming the software development process by offering real-time code suggestions. These tools promise to improve the productivity by reducing cognitive load and speeding up task completion. Previous exploratory studies, however, show that developers sometimes perceive the automatic suggestions as intrusive. As a result, they feel like their productivity decreased. Objective We propose two theories on the impact of automatic suggestions on frustration and productivity. First, we hypothesize that experienced developers are frustrated from automatic suggestions (mostly from irrelevant ones), and this also negatively impacts their productivity. Second, we conjecture that novice developers benefit from automatic suggestions, which reduce the frustration caused from being stuck on a technical problem and thus increase their productivity. Method We conducted a quasi-experimental study with 33 developers (16 experts and 17 novices). Each participant performed two Java maintenance tasks under two interaction modes: automatic suggestions enabled (proactive) and automatic suggestions disabled (reactive, on-demand). We collected IDE-level interaction traces via a custom plugin, post-task questionnaires, and physiological signals ( i.e., heart rate variability) via a wearable device. We use statistical hypothesis tests and models to study the effects of the treatments ( i.e., automatic suggestions enabled/disabled) on the outcomes (frustration and productivity). Results The aim of our work is to corroborate or contradict our theories with empirical evidence collected in a quasi-experimental study. Conclusions The results corroborate both our experience-dependent theories. Novice developers benefit from proactive suggestions, showing lower frustration and higher productivity when automatic suggestions are enabled. Instead, expert developers benefit from reactive, on-demand interaction: disabling automatic suggestions is associated with lower frustration and higher productivity. Our findings motivate adaptive interaction strategies that tailor proactivity to developer experience, balancing proactive guidance for novices with control-preserving, user-invoked assistance for experts.