Attention is My Memory: Positional Encoding-free Length Invariance in Transformers
This paper introduces a positional encoding-free transformer augmented with a single content-addressed memory from DABSN, enabling extreme length generalization without positional embeddings or context-index limits. Trained on 256-token sequences, the model achieves 96.9% accuracy on multi-query associative recall at 64× training length, 100% accuracy at 8× memory load, and 96–100% passkey retrieval accuracy at one million tokens (4096× training length). The approach also matches positional transformers on character-level language modeling at training length. Across 459 experimental runs, the study demonstrates how a lightweight memory interface can overcome major limitations of positional addressing in transformer-based retrieval.
Authors
- Nicholas Rosdahl
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24268810
- Primary Topic
- Topic Modeling
- Type
- preprint