Streaming Is Not Memory: Constant-State Born-Attention Byte Models on PG-19

A language model whose attention keeps a fixed-size state can read an input of any length in one pass, with memory that does not grow with the input. That property is often reported as a long context. We separate it into three claims and test each one on byte-level PG-19. The models are fully complex-valued, built in the CNet framework, and use linear Born attention, | |^2 evaluated through a density-matrix state, with a fixed per-head decay and rotary positions. A model with 12M complex parameters (23.9M real-scalar equivalents) trained in strict FP32 on half a pass of the full 11.7 GB corpus scores 1.312 bits per byte on the PG-19 test partition when each 5,242,880-byte window is read in one unbroken pass with 12.8 MB of state; the MBLM hierarchy reports 2.448 at that window length, an uncontrolled comparison, and 1.240 at 98k bytes. Streaming is stable, but we find little aggregate benefit from persistence beyond a few thousand bytes. A reset sweep on identical validation bytes shows that carried state helps by 0.012 bits per byte for the 12M model and 0.006 for a 4M model, and both curves follow a fixed-cost-per-reset model, ΔL(C) = ΔL(∞)(1 − 4096/C), with residuals below 4e-5. This is consistent with most of the gain coming from state recovery after a reset. Extending the reset interval beyond 256k bytes improves the loss by about 1.8e-4 bits per byte, as the fixed-cost model predicts; the residual is below 4e-5, but an aggregate-loss experiment does not isolate the contribution of long-range information, and retrieval was not measured. The fixed decay suppresses old contributions exponentially, with half-lives of 22 to 709 bytes per head. A real-valued Transformer with the same parameter count, bytes and schedule, scored on the same windows, reaches 1.245 bits per byte on test against 1.324 for the Born model with the same 4,096-byte resets. In a matched five-configuration experiment at 10,000 steps in PyTorch, the same complex Born design achieves the lowest test loss, 1.413 against 1.495 for the Transformer, while Born attention placed in the real Transformer costs 0.017: the relative performance of Born and softmax depends on the representation and read-out around them. CNet's default initialization, which we show to be poorly conditioned, accounts for 0.023 of the original gap: with a Gaussian start the same design reaches 1.301 after 60,000 steps, and the Transformer still leads at full length (1.245). The ranking depends on the training budget: the complex Born design is better with a 10,000-step schedule and worse with a 60,000-step one. Constant-state streaming is a property of the method; long effective memory is not; neither is quality at equal size at full training length. Preprint, October 2026. Companion to "Born Attention is Linear" (doi:10.5281/zenodo.23685041). Code: the open-source CNet framework.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-27
DOI
https://doi.org/10.5281/zenodo.23685936
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Streaming Is Not Memory: Constant-State Born-Attention Byte Models on PG-19

Marcel Crâşmaru
5 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Streaming Is Not Memory: Constant-State Born-Attention Byte Models on PG-19

Marcel Crâşmaru
preprint en

Abstract

A language model whose attention keeps a fixed-size state can read an input of any length in one pass, with memory that does not grow with the input. That property is often reported as a long context. We separate it into three claims and test each one on byte-level PG-19. The models are fully complex-valued, built in the CNet framework, and use linear Born attention, | |^2 evaluated through a density-matrix state, with a fixed per-head decay and rotary positions. A model with 12M complex parameters (23.9M real-scalar equivalents) trained in strict FP32 on half a pass of the full 11.7 GB corpus scores 1.312 bits per byte on the PG-19 test partition when each 5,242,880-byte window is read in one unbroken pass with 12.8 MB of state; the MBLM hierarchy reports 2.448 at that window length, an uncontrolled comparison, and 1.240 at 98k bytes. Streaming is stable, but we find little aggregate benefit from persistence beyond a few thousand bytes. A reset sweep on identical validation bytes shows that carried state helps by 0.012 bits per byte for the 12M model and 0.006 for a 4M model, and both curves follow a fixed-cost-per-reset model, ΔL(C) = ΔL(∞)(1 − 4096/C), with residuals below 4e-5. This is consistent with most of the gain coming from state recovery after a reset. Extending the reset interval beyond 256k bytes improves the loss by about 1.8e-4 bits per byte, as the fixed-cost model predicts; the residual is below 4e-5, but an aggregate-loss experiment does not isolate the contribution of long-range information, and retrieval was not measured. The fixed decay suppresses old contributions exponentially, with half-lives of 22 to 709 bytes per head. A real-valued Transformer with the same parameter count, bytes and schedule, scored on the same windows, reaches 1.245 bits per byte on test against 1.324 for the Born model with the same 4,096-byte resets. In a matched five-configuration experiment at 10,000 steps in PyTorch, the same complex Born design achieves the lowest test loss, 1.413 against 1.495 for the Transformer, while Born attention placed in the real Transformer costs 0.017: the relative performance of Born and softmax depends on the representation and read-out around them. CNet's default initialization, which we show to be poorly conditioned, accounts for 0.023 of the original gap: with a Gaussian start the same design reaches 1.301 after 60,000 steps, and the Transformer still leads at full length (1.245). The ranking depends on the training budget: the complex Born design is better with a 10,000-step schedule and worse with a 60,000-step one. Constant-state streaming is a property of the method; long effective memory is not; neither is quality at equal size at full training length. Preprint, October 2026. Companion to "Born Attention is Linear" (doi:10.5281/zenodo.23685041). Code: the open-source CNet framework.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon