Mean Answer Loss Overstates What a Language Model Knows: Measure Retrieval on the First Token, Against a Wrong Question

Studies of how pretraining data shapes factual knowledge commonly score a fact by the mean token loss of its gold answer under a question. For multi-token answers this mean mixes two different things: retrieving the answer from the question, which shows up on the answer's first token, and completing the answer string once its first token is given, which shows up on the rest and is learned from the string alone. A one-line control separates them: score the same answer under the correct question and under another fact's question, and read the first-token difference. We show the size of the confusion on both sides. On a from-scratch bench, ten final checkpoints from six training cells that had shown clean, repeatable dose response under the mean measure had learned only string completion: the first token never moved at any dose, and the wrong question did as well as the right one. Redesigned so that each answer starts with a unique token, the same bench retrieves, with a dose curve and a ceiling that the mean measure would not have shown. On 18 public models from five labs the correct question lowers the first token by 1.5 to 7.0 nats and wins on 71 to 93% of facts, and the gain grows with size in every family. On the Pythia suite, the mean loss improves with pretraining co-occurrence count from the lowest counts at every size, while exact-match accuracy and the first-token loss agree that nothing is learned below about a hundred co-occurrences at any size. We recommend reporting the first-token loss under a counterfactual question beside any mean-loss measure of factual knowledge.