Comparative Structured Observation

Although HCI researchers often generate and compare new design concepts, they lack an established method for rigorously conducting qualitative assessments . We define and characterize Comparative Structured Observation as a qualitative research method that takes advantage of the structure of controlled experiments to generate comparable, ecologically relevant experiences with two or more design variants, often implemented as medium-fidelity prototypes. Researchers observe users and ask them to compare and reflect on each variant. We identify criteria for creating a successful Comparative Structured Observation study and illustrate variations of the method by analyzing four published studies. We also examine six additional studies (three “near” and three “far”) to clarify the boundary between what should and should not be considered a Comparative Structured Observation . We discuss the benefits and limitations of the method and argue that gathering comparative reflections about design variants can help researchers assess and advance their design concepts.