How Far Can Agents Get by Writing a Language Model's Weights? A pre-registered measurement against SGD on TinyStories
Frontier language-model agents can now write the weights of small neural networks directly, with no optimizer. We ask how much of what SGD learns they can replace on natural text. Opus 5.5 agents wrote every weight of a one-layer attention-only transformer language model on TinyStories (vocabulary 4,096, d_model = 256, 8 heads), one named mechanism per agent episode, from closed-form count statistics of the training split and without a single gradient step or loss evaluation. We measure the gap closed, G = (L_bigram − L_written)/(L_bigram − L_SGD), between a bigram-in-weights floor and SGD on the same architecture, on a sealed split that no writer saw. The design, the thresholds and the stopping rules were pre-registered. The confirmatory hypothesis (G ≥ 0.5) fails: the final written model reaches a sealed loss of 3.581 nats against 3.904 for the floor and 2.079 for SGD, G = 0.177. The writers' predictions of their own components' contributions, written down before any measurement, largely rank the measured contributions correctly (Spearman ρ = 0.833, 95% CI 0.526–0.961, n = 20). A writer batch with 50 forward passes of feedback per writer reached G = 0.270 (descriptive), and G = 0.295 when each pass ran the writer's weights inside the working M1/L1 model (exploratory). An exploratory analysis on dev finds the remaining gap concentrated on mid-frequency tokens (52% of it) and on tokens new to the story (58%), where the written model closes almost nothing (G = 0.026); part of this comes from a direct path that the residual-stream budget limits to rank 123. The agents write a count model with working copy, agreement and structure heads; they do not write the prediction of new content from context. Contents of this record. The preprint (PDF); code and configuration; the pre-registration (prereg/PREREG_A.md) with its dated amendments A1–A4; the decision log; the record files (every launch, episode label, integration, milestone announcement and look at held-out data); the written models and their mechanism notes; and the scripts that regenerate every number in the paper (paper/scripts/make_numbers.py, commit 149048a). Writer transcripts are not released; per-episode labels, audit summaries, final answers and submissions are. The writer harness is a locally modified fork of OpenWiSH (changes listed in harness/VENDORED.md).
Authors
- Xintian Huang (ORCID: https://orcid.org/0009-0000-7664-7108)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24423263
- Primary Topic
- Topic Modeling
- Type
- preprint