Relation Summaries for Dynamic Attention Head Routing: A Matched-Budget Study of Parity and Key-Value Retrieval

This study tests whether information from attention heads within a Transformer layer helps a router decide how many additional heads to activate at a matched average head budget. The four-layer causal Transformer uses two always-on heads. The router can use their outputs and attention-weighted summaries of pairwise relations carried across layers. On the four-clue parity task, switching from fixed four-head routing to variable head counts raises mean test accuracy from 82.83% to 91.85%, with 3.988 active heads per query position on average. Removing the relation summary lowers accuracy in all three paired training seeds. On the key-value retrieval task, all three configurations use variable head counts and average approximately four active heads per query position. Across eight paired seeds, mean four-key exact-match accuracy is 89.02% with hidden states alone, 76.32% after adding the always-on head outputs, and 89.00% after also adding the relation summary. The additional routing inputs do not yield a consistent improvement over hidden-state routing on this task. Both tasks are synthetic and use a 128-token context. The implementation computes all candidate heads before applying hard masks, so the budget measures the number of active heads. Runtime and memory gains remain unmeasured.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24213751
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Relation Summaries for Dynamic Attention Head Routing: A Matched-Budget Study of Parity and Key-Value Retrieval

Zhongren Wang
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Relation Summaries for Dynamic Attention Head Routing: A Matched-Budget Study of Parity and Key-Value Retrieval

Zhongren Wang
preprint en

Abstract

This study tests whether information from attention heads within a Transformer layer helps a router decide how many additional heads to activate at a matched average head budget. The four-layer causal Transformer uses two always-on heads. The router can use their outputs and attention-weighted summaries of pairwise relations carried across layers. On the four-clue parity task, switching from fixed four-head routing to variable head counts raises mean test accuracy from 82.83% to 91.85%, with 3.988 active heads per query position on average. Removing the relation summary lowers accuracy in all three paired training seeds. On the key-value retrieval task, all three configurations use variable head counts and average approximately four active heads per query position. Across eight paired seeds, mean four-key exact-match accuracy is 89.02% with hidden states alone, 76.32% after adding the always-on head outputs, and 89.00% after also adding the relation summary. The additional routing inputs do not yield a consistent improvement over hidden-state routing on this task. Both tasks are synthetic and use a 128-token context. The implementation computes all candidate heads before applying hard masks, so the budget measures the number of active heads. Runtime and memory gains remain unmeasured.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon