Attention Is All We Had — But Not What We Needed: Convergent State Machine for Iterative Energy-Based Language Generation

We introduce the Convergent State Machine (CSM), a novel architecture for language generation that replaces attention entirely with energy-based iterative state refinement. A 150M parameter CSM with zero attention layers matches GPT-2 1.5B on MMLU within 0.3%, using 10x fewer parameters and 13x less training data. State convergence dynamics scale with model size: 66M converges by iteration 15, 150M sustains refinement to iteration 30+. Total training compute: under $50 of A100 GPU time. Keywords: attention-free, iterative reasoning, energy-based, state machine, language model, scaling law