Beyond the Reach of Gradients: LLM Agents Write Exact Weights Where Gradient Descent Fails

Gradient descent finds solutions that its update path can reach; a large language model (LLM) that writes weights directly needs no such path. We test whether this difference can be measured, using two constrained architectures where gradients have little to work with: a transformer with ternary weights, Heaviside activations and hard attention (F0), and an integer leaky-integrate-and-fire spiking network (S0). Under a fixed protocol (no training, no forward passes of the network before scoring, and exhaustive or fresh-seed scoring), Claude Opus writes exact F0 networks for all 9 tasks we set (45/45 attempts). Gradient training finds no exact F0 solution on 5 of the 6 tasks where an unconstrained transformer trains (0 of 211 unassisted and 0 of 190 size-matched runs); on the sixth, sparse parity, 3 of 50 unassisted runs are exact but about 880× larger than the Writer's, and size-matched training is 0/150. On 10 random tasks with no published construction, Opus writes 10/10 exact networks, against 0/150 for F0 gradient training. On S0, all 22 written networks (11 by Opus, 11 by an open-weight model) are exact; 20 are mechanically proven correct for every input length and the other two for every length below 2³¹ − 22. Surrogate-gradient training, which sees lengths up to 10, finds no exact network on the 7 tasks that need more than a few states (0/397). The written networks are the shortest we have under a decodable code: 3.5–168× shorter than compiled ones, and 9–377× shorter than the exact gradient-trained ones. An open-weight model reproduces correctness on every architecture; on F0 its networks are usually much larger. We do not claim that these networks are untrainable in an absolute sense, that only LLMs can find them, or that the effect scales.