Predicting reference matches from generator and reader states

Predicting reference matches from generator and reader states: Stage P exploratory pilot report (Version 1.0) A published pilot study reported that a language model's hidden state predicts its own eventual answer better than a second model reading the same text. This report tests a related question on factual recall: can a model's own state predict which of its sampled answers a reference matcher will reject, better than an outside reader can? Qwen2.5-1.5B-Instruct generated 16 justification-then-answer responses to each of 120 factual questions in the T-08R powered ladder. The analysis uses the 999 committed generations on the 64 questions that produced both matching and non-matching answers. A linear probe read the generator's state (Self) and a second probe read Qwen2.5-3B-Instruct on the identical text (Reader), at a depth fixed before fitting: block 14 of 28 and block 18 of 36. Probes were scored on held-out questions by within-question pooled AUC. The smallest meaningful Self advantage, +0.03 AUC, was fixed before any probe was fitted. At the fixed depth, the paired 95% intervals rule out a +0.03 Self advantage at the eighth justification token (Self minus Reader −0.008, interval −0.036 to +0.019) and at the end of the justification (−0.035, interval −0.067 to −0.002). Immediately before the formal answer the difference was +0.013 (interval −0.025 to +0.051), so an advantage of that size remains open there. The same three conclusions hold after removing the 24 generations on which two string matchers disagreed. Adding the Self score to the generator's own output probabilities did not measurably improve prediction. A descriptive layer sweep shows that the fixed depth sits just before predictive signal appears in both models. In later layers the larger Reader scores at least as high as Self. That sweep was not pre-specified and has no intervals. Scope and limits. This is an exploratory pilot on one model pair and 64 questions. The endpoint is a string matcher's verdict, and independent validation of factual correctness remains open. Historical token sequences were reconstructed from saved text and screened by token count. A confirmatory study with fresh questions and a comparable-size reader is planned and has produced no data. Files. The report (PDF) and Stage_P_Reproducibility.zip, which contains the held-out predictions, per-question win counts, fold assignments, run configuration, the analysis notebook, the dated pre-fit amendment and layer curves. Running python reproduce_tables.py rebuilds both report tables on a CPU without model inference. A full probe refit additionally requires the 2.8 GB state capture, which is not included. Related records. T-08R ladder data: 10.5281/zenodo.21707044. Ladder report: 10.5281/zenodo.21714854.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-21
DOI
https://doi.org/10.5281/zenodo.24412306
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Predicting reference matches from generator and reader states

Joshua A. Duran
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Predicting reference matches from generator and reader states

Joshua A. Duran
preprint en

Abstract

Predicting reference matches from generator and reader states: Stage P exploratory pilot report (Version 1.0) A published pilot study reported that a language model's hidden state predicts its own eventual answer better than a second model reading the same text. This report tests a related question on factual recall: can a model's own state predict which of its sampled answers a reference matcher will reject, better than an outside reader can? Qwen2.5-1.5B-Instruct generated 16 justification-then-answer responses to each of 120 factual questions in the T-08R powered ladder. The analysis uses the 999 committed generations on the 64 questions that produced both matching and non-matching answers. A linear probe read the generator's state (Self) and a second probe read Qwen2.5-3B-Instruct on the identical text (Reader), at a depth fixed before fitting: block 14 of 28 and block 18 of 36. Probes were scored on held-out questions by within-question pooled AUC. The smallest meaningful Self advantage, +0.03 AUC, was fixed before any probe was fitted. At the fixed depth, the paired 95% intervals rule out a +0.03 Self advantage at the eighth justification token (Self minus Reader −0.008, interval −0.036 to +0.019) and at the end of the justification (−0.035, interval −0.067 to −0.002). Immediately before the formal answer the difference was +0.013 (interval −0.025 to +0.051), so an advantage of that size remains open there. The same three conclusions hold after removing the 24 generations on which two string matchers disagreed. Adding the Self score to the generator's own output probabilities did not measurably improve prediction. A descriptive layer sweep shows that the fixed depth sits just before predictive signal appears in both models. In later layers the larger Reader scores at least as high as Self. That sweep was not pre-specified and has no intervals. Scope and limits. This is an exploratory pilot on one model pair and 64 questions. The endpoint is a string matcher's verdict, and independent validation of factual correctness remains open. Historical token sequences were reconstructed from saved text and screened by token count. A confirmatory study with fresh questions and a comparable-size reader is planned and has produced no data. Files. The report (PDF) and Stage_P_Reproducibility.zip, which contains the held-out predictions, per-question win counts, fold assignments, run configuration, the analysis notebook, the dated pre-fit amendment and layer curves. Running python reproduce_tables.py rebuilds both report tables on a CPU without model inference. A full probe refit additionally requires the 2.8 GB state capture, which is not included. Related records. T-08R ladder data: 10.5281/zenodo.21707044. Ladder report: 10.5281/zenodo.21714854.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon