Attention Is All We Had — But Not What We Needed: Convergent State Machine for Iterative Energy-Based Language Generation

We introduce the Convergent State Machine (CSM), a novel architecture for language generation that replaces attention entirely with energy-based iterative state refinement. Three models trained (66M, 150M, 331M), all with zero attention layers. CSM 150M matches GPT-2 1.5B on MMLU within 0.4%, using 10x fewer parameters and 13x less data. Key finding: hard problems show 60% perplexity improvement with more iterations while easy problems degrade 60%. The model reasons deeper on harder problems — not repeated computation, but genuine difficulty-dependent reasoning. Iteration scaling confirmed across three model sizes: 66M converges at iter 15, 150M at iter 30, 331M at iter 40. Total training compute: under $100 of A100 GPU time.