Zero-Interference Associative Recall in a Single Linear Recurrent Language Model: Frozen Stable Keys, a Trigger Gate, and Decoupled Training
Autoregressive language modeling must do two things at once: compress a mostly-local context into a state that drives fluent next-token prediction, and, on occasion, re-encounter a previously seen key and emit the exact value bound to it. Linear-time recurrent and state-space models do the first at constant per-step cost but recall a far-away value poorly; attention recalls exactly but is quadratic. A common resolution attaches a separate memory or retrieval module. We challenge that partition with an attention-free linear backbone of Mamba2-style gated blocks and MLP blocks, into which we splice a frozen-key, trigger-gated associative readout addressed by stable token keys (orthogonal by default). The readout contributes nothing to the language-model stream: on held-out WikiText-103, perplexity measured on identical sequences with the readout on and off is identical (Δ = +0.0 on validation 48.37 and test 56.60), while multi-query associative recall is exact (1.000 at nearest, mid, and farthest distance) within the tested regime (store size ≤ 256, stable-key dimension 256, marker-token required). Against matched baselines a Mamba-2-style SSM, a no-forgetting linear-attention accumulator, and a full-attention Transformer all recover none of the stored pairs. We further test the necessity of the orthogonal key family: frozen random keys are equally exact at load one, so orthogonality is a sufficient default for near-key discrimination rather than a requirement, and we give the first direct measurements of the dk=256 model past load one (loads 2–4: nearest and mid recall remain exact, degradation concentrates on the oldest binding). This is a proof of principle at the load-one boundary with an explicit trigger signal, at a single model scale. Version 3 is a major revision of 10.5281/zenodo.22243420. The title and framing are changed to state the claim scope explicitly (zero-interference coexistence under frozen stable keys, a trigger gate, and decoupled training, at the load-one boundary). New experimental content in this version: (i) a controlled ablation showing that the orthogonal key family is a sufficient default rather than a necessary condition for exact recall at load one (frozen random keys are equally exact); and (ii) the first direct measurements of the dk=256 model past load one (loads 2–4), showing that nearest and mid-distance recall remain exact while degradation concentrates on the oldest binding. We also add per-seed robustness tables, a plain-language motivation, an expanded reference list, and precise statements of the tested regime (store size ≤ 256, stable-key dimension 256, marker token required, single model scale). This record continues the author's series on content-addressed recursive state: 10.5281/zenodo.22184105 (v1) and 10.5281/zenodo.22243420 (v2).
Authors
- Ziheng Zhou (ORCID: https://orcid.org/0000-0002-7080-8561)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-09-07
- DOI
- https://doi.org/10.5281/zenodo.22184104
- Primary Topic
- Handwritten Text Recognition Techniques
- Type
- preprint