Kernel, Transport, Support: A Unified Prescription for Attention, State-Space and Delta-Rule Sequence Mixers
Softmax attention, sliding-window attention, linear attention, state-space models and delta-rule recurrences such as Gated DeltaNet and Kimi Delta Attention are usually presented as separate families. This note writes them in one parameterization: a kernel (bilinear or exponential), a transport given by products of per-step transitions, and a set of kept tokens or memory slots, all driven by local queries, keys, values and gates. We separate exact re-realizations of the same input–output map from structural differences. Re-realizations are evaluation order, primal versus dual state, and time-dependent changes of state basis; the familiar "gauge" that turns complex diagonal recurrences into rotary embeddings is a special case. Structural differences are described by a signature: the kernel, the write rank, the type of the transitions and the type of support. A signature is a description, not an invariant, and separations between function classes are cited, not derived. A gauge depending only on position turns fixed transports into positional features. For data-dependent transports, a gauge relocates the transport into prefix-dependent features; it does not remove it. As a worked example we compile the known coordinate forms of the memory of sliding-window attention, a delay line of length W: shift register, ring buffer, and a Fourier form that is a single tied delta-rule step with unit-modulus complex gates. With real gates, such a step has Jordan chains of length at most three, which follows from known indefinite-inner-product arguments. We tabulate the layer composition of eleven released models and close with design heuristics, each stated with its evidence. The note proves no new theorem; it aims at a precise, attributed common language for these models.
Authors
- Yan Ru Pei (ORCID: https://orcid.org/0000-0002-6999-3691)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-13
- DOI
- https://doi.org/10.5281/zenodo.24191078
- Primary Topic
- Topic Modeling
- Type
- preprint