From Quadratic Attention to Selective Recurrent Memory (A Mathematical and Architectural Study of the Path from GPT-2 to Kimi K3)

Large language model development is frequently summarized as a scaling process in which parameter counts, training data, and accelerator budgets increase together. This view is incomplete. A second line of progress concerns the structure of memory and information flow inside the model. This tutorial survey develops a unified mathematical account of that evolution, from softmax self-attention and the autoregressive key-value cache to linear attention, DeltaNet, Gated DeltaNet, Kimi Delta Attention, Multi-head Latent Attention, Attention Residuals, and sparse Mixture-of-Experts. Kimi K3 is used as a contemporary case study because its 2026 architecture combines these mechanisms at multi-trillion-parameter scale. The Kimi Team reports approximately 2.8 trillion total parameters, 104 billion activated parameters, a one-million-token training context, a 3:1 alternation of Kimi Delta Attention and Gated MLA, and Stable LatentMoE with 896 routed experts of which 16 are selected per token. We derive the principal recurrence equations, explain their memory interpretation, compare sequence and cache complexity, and provide reproducible low-dimensional numerical examples. The central pedagogical contribution is a three-axis view of modern model design: information flow across sequence length, network depth, and model width. This article does not claim new benchmark results. It provides an original synthesis, derivation-oriented tutorial, and reproducible teaching examples intended for graduate-level study of current LLM architectures.