Your Query is My Key: Halving Attention Parameters via Cross-Head Cyclic Rings

Query and key projections in self-attention enter the score through their product, which creates a continuous redundancy in their parameterisation. Tying the projections removes this freedom but restricts the score matrix. This paper derives the corresponding gradient identity, describes the symmetry and sign constraints of direct tying, and introduces a cross-head cyclic construction that uses one projection per head and forms scores from neighbouring projections in a ring. The query-key parameter count is reduced by half. Experiments cover associative recall, character-level language modelling, memory use, and small-scale expressiveness tests. In several tested settings, the cyclic model reaches high task accuracy while using half the query-key parameters. The measurements do not establish a general advantage, but they show that the cyclic restriction can retain strong performance at the tested scale

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24377782
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Your Query is My Key: Halving Attention Parameters via Cross-Head Cyclic Rings

Sidharth S S
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
article

Your Query is My Key: Halving Attention Parameters via Cross-Head Cyclic Rings

Sidharth S S
article en

Abstract

Query and key projections in self-attention enter the score through their product, which creates a continuous redundancy in their parameterisation. Tying the projections removes this freedom but restricts the score matrix. This paper derives the corresponding gradient identity, describes the symmetry and sign constraints of direct tying, and introduces a cross-head cyclic construction that uses one projection per head and forms scores from neighbouring projections in a ring. The query-key parameter count is reduced by half. Experiments cover associative recall, character-level language modelling, memory use, and small-scale expressiveness tests. In several tested settings, the cyclic model reaches high task accuracy while using half the query-key parameters. The measurements do not establish a general advantage, but they show that the cyclic restriction can retain strong performance at the tested scale

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon