Your Query is My Key: Halving Attention Parameters via Cross-Head Cyclic Rings
Query and key projections in self-attention enter the score through their product, which creates a continuous redundancy in their parameterisation. Tying the projections removes this freedom but restricts the score matrix. This paper derives the corresponding gradient identity, describes the symmetry and sign constraints of direct tying, and introduces a cross-head cyclic construction that uses one projection per head and forms scores from neighbouring projections in a ring. The query-key parameter count is reduced by half. Experiments cover associative recall, character-level language modelling, memory use, and small-scale expressiveness tests. In several tested settings, the cyclic model reaches high task accuracy while using half the query-key parameters. The measurements do not establish a general advantage, but they show that the cyclic restriction can retain strong performance at the tested scale
Authors
- Sidharth S S (ORCID: https://orcid.org/0009-0007-4442-156X)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24377782
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00