An Instrument for Observing Teams to Explicate Regulation Strategies in Computer Science

Motivation and Objectives . Informed by a long tradition of studying regulation strategies in general education, recent work in Computing Education Research has highlighted the importance and effects of such strategies in computing-related classrooms as well. Most of this work has focused on self-regulation of individuals or dyads, for example, in pair programming. Little, however, is known about regulation in larger groups which naturally occur in, for example, active-learning classrooms, Software Engineering lab courses, or capstone projects. Moreover, research involving regulation strategies, including, but not limited to computing contexts, predominantly relies on self-reported data. While recent work has advocated using trace data from learning management systems to assess self-regulation strategies more objectively, such systems cannot capture regulation as it occurs in many versions of group work, for example, in group discussions and interactions in a classroom. We thus aim to develop an observation instrument that can help researchers to explicate regulation strategies in Computer Science and, more generally, computing-related group work. Together with self-reported data and—where applicable—trace data, such an instrument would facilitate obtaining a more complete picture of where and how regulation strategies emerge and develop. Methods . Using data from semi-structured interviews with students from two capstone projects, we conducted a deductive qualitative analysis to add concrete descriptors to Miller and Hadwin’s framework for regulated learning. We refined the resulting coding scheme into an observation instrument that was field-tested for saturation, ease-of-use, and reliability in a Software Engineering lab course. Results . Our main result is OTTERS , an instrument for observing teams to explicate regulation strategies in computing contexts. Initial field-testing indicates that this instrument, despite its granularity, is easy to use and reliable. Furthermore, first proof-of-concept observations lead to patterns of regulation activities that are clearly distinguishable, thus suggesting construct validity. Discussion . Whereas previous work focused on self-reported data or trace data obtained from learning management systems, our observation instrument adds another facet to assessing regulation strategies. Compared to self-reported data, it trades off the bias of possibly unreliable self-assessment and the limitations of external observations. Compared to analyses based on high-resolution, objective log data, it yields observations on a coarser granularity but is applicable not only to self-regulation but to socially shared regulation and co-regulation as well. Our observation instrument thus complements the current methodological portfolio of assessing regulation strategies in Computer Science and related contexts.