Model, Not Mode: A Pre-Registered Model x Mode Grid on a Constrained Design Task

Chat apps let users pick a model and how hard it thinks. We asked which choice matters more on an open-ended, constrained task: designing a local AI workstation for a 32 GB machine. We crossed six model families with two modes, fast and reasoning, and collected five answers per cell in each vendor’s app (60 answers). Three blind raters, one human and two LLMs, scored every answer on a nine-dimension rubric. A pre-registered variance decomposition found that family explained far more variance in the total score than mode (delta omega squared −0.527, 95% CI [−0.666, −0.417]). Reasoning scored 1.37 points higher out of 18 (95% CI [0.77, 1.93]), but most of the gain came from one family and halved once answer length was removed. The secondary hypotheses were exploratory because rater agreement fell below the bar on seven of nine dimensions. Comparisons should name the model first, and still report the mode, because its effect differs between models.