Getting It Right Is Not Getting It Right the Same Way: Why the Evaluation of Inference in Language Models Requires Process Measures

Inference and commonsense reasoning in language models are evaluated almost entirely by whether the model's output matches a gold label. This is an outcome measure, and outcome measures have a structural limitation that better items cannot repair: they cannot distinguish a system that performed an inference from a system that retrieved or matched a pattern which rendered the inference unnecessary, because both routes terminate in the same response. We argue that this limitation explains a persistent pattern in the field, in which benchmark scores rise steeply while the failures those benchmarks were built to detect continue to appear, and we argue that it cannot be resolved by adversarial filtering, contamination auditing, or the inspection of generated reasoning traces, each of which remains a function of the model's output. Research on human reading offers a different class of measurement. Reading time, event-related potentials, and hemodynamic response index the cost of bridging, integration, and world-knowledge retrieval while comprehension unfolds, before and independently of whether the reader ultimately answers correctly. We specify what a process-level comparison between reader and model would require, including matched materials, difficulty matched on word frequency and contextual predictability, and matched item sets in which inference is necessary versus items in which it can be bypassed through a surface cue. We then state the failure modes directly. Predictive alignment between model representations and neural signals is not evidence of shared mechanism, and we treat this as a premise rather than a limitation. We close with implications for reporting practice and with a hypothesis, not a finding, about data efficiency.