Token uniformity in residual self-attention at random initialization comes from the mean token

We study residual self-attention networks X_(l+1) = X_l+α_l A_l X_l W_l at initialization, with residual scales α_l ≥ 0. The value matrices W_l are Gaussian. The attention matrices A_l are row-stochastic and may depend on the tokens and on any randomness independent of the value matrices, which covers softmax attention with query and key matrices independent of the value matrices. Let S_l be the squared Frobenius norm of X_l after the mean token is subtracted from every token. If the tokens have dimension d ≥ 2 and the input X_0 is deterministic with S_0>0, then log S_l is a submartingale for every choice of attention and of residual scales; this can fail for d = 1. Consequently, for a network of depth L with α_l = c/√L, the ratio S_L/‖X_L‖_F² stays bounded below in probability, uniformly in L and in the attention. At a fixed scale α it decays at most at the exponential rate log (1+α²n), where n is the number of tokens.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24326593
Primary Topic
Stochastic Gradient Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Token uniformity in residual self-attention at random initialization comes from the mean token

Yitzchak Shmalo
2 links
Zenodo (CERN European Organization for Nuclear Research)
Stochastic Gradient Optimization Techniques
preprint

Token uniformity in residual self-attention at random initialization comes from the mean token

Yitzchak Shmalo
preprint en

Abstract

We study residual self-attention networks X_(l+1) = X_l+α_l A_l X_l W_l at initialization, with residual scales α_l ≥ 0. The value matrices W_l are Gaussian. The attention matrices A_l are row-stochastic and may depend on the tokens and on any randomness independent of the value matrices, which covers softmax attention with query and key matrices independent of the value matrices. Let S_l be the squared Frobenius norm of X_l after the mean token is subtracted from every token. If the tokens have dimension d ≥ 2 and the input X_0 is deterministic with S_0>0, then log S_l is a submartingale for every choice of attention and of residual scales; this can fail for d = 1. Consequently, for a network of depth L with α_l = c/√L, the ratio S_L/‖X_L‖_F² stays bounded below in probability, uniformly in L and in the attention. At a fixed scale α it decays at most at the exponential rate log (1+α²n), where n is the number of tokens.

Zenodo (CERN European Organization for Nuclear Research)
Hebrew University of Jerusalem (IL)
Stochastic Gradient Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon