Kernel, Transport, Support: A Unified Prescription for Attention, State-Space and Delta-Rule Sequence Mixers

Softmax attention, sliding-window attention, linear attention, state-space models and delta-rule recurrences such as Gated DeltaNet and Kimi Delta Attention are usually presented as separate families. This note writes them in one parameterization: a kernel (bilinear or exponential), a transport given by products of per-step transitions, and a set of kept tokens or memory slots, all driven by local queries, keys, values and gates. We separate exact re-realizations of the same input–output map from structural differences. Re-realizations are evaluation order, primal versus dual state, and time-dependent changes of state basis; the familiar "gauge" that turns complex diagonal recurrences into rotary embeddings is a special case. Structural differences are described by a signature: the kernel, the write rank, the type of the transitions and the type of support. A signature is a description, not an invariant, and separations between function classes are cited, not derived. A gauge depending only on position turns fixed transports into positional features. For data-dependent transports, a gauge relocates the transport into prefix-dependent features; it does not remove it. As a worked example we compile the known coordinate forms of the memory of sliding-window attention, a delay line of length W: shift register, ring buffer, and a Fourier form that is a single tied delta-rule step with unit-modulus complex gates. With real gates, such a step has Jordan chains of length at most three, which follows from known indefinite-inner-product arguments. We tabulate the layer composition of eleven released models and close with design heuristics, each stated with its evidence. The note proves no new theorem; it aims at a precise, attributed common language for these models.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24191078
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Kernel, Transport, Support: A Unified Prescription for Attention, State-Space and Delta-Rule Sequence Mixers

Yan Ru Pei
1 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Kernel, Transport, Support: A Unified Prescription for Attention, State-Space and Delta-Rule Sequence Mixers

Yan Ru Pei
preprint en

Abstract

Softmax attention, sliding-window attention, linear attention, state-space models and delta-rule recurrences such as Gated DeltaNet and Kimi Delta Attention are usually presented as separate families. This note writes them in one parameterization: a kernel (bilinear or exponential), a transport given by products of per-step transitions, and a set of kept tokens or memory slots, all driven by local queries, keys, values and gates. We separate exact re-realizations of the same input–output map from structural differences. Re-realizations are evaluation order, primal versus dual state, and time-dependent changes of state basis; the familiar "gauge" that turns complex diagonal recurrences into rotary embeddings is a special case. Structural differences are described by a signature: the kernel, the write rank, the type of the transitions and the type of support. A signature is a description, not an invariant, and separations between function classes are cited, not derived. A gauge depending only on position turns fixed transports into positional features. For data-dependent transports, a gauge relocates the transport into prefix-dependent features; it does not remove it. As a worked example we compile the known coordinate forms of the memory of sliding-window attention, a delay line of length W: shift register, ring buffer, and a Fourier form that is a single tied delta-rule step with unit-modulus complex gates. With real gates, such a step has Jordan chains of length at most three, which follows from known indefinite-inner-product arguments. We tabulate the layer composition of eleven released models and close with design heuristics, each stated with its evidence. The note proves no new theorem; it aims at a precise, attributed common language for these models.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon