The Geometric and Algebraic Structure of Transformer Attention

This paper develops a unified geometric and algebraic analysis of the attention mechanism in Transformer architectures. The central observation is that, given a token embedding matrix $X\in R^{N\times d}$, every similarity matrix producible by a single attention head lies inside a nested hierarchy $\mathcal{V}(d_k)(\mathbf{X}) \subset \mathcal{V} (\mathbf{X}) \subset \mathbb{R}^{N \times N}$ is a vector subspace of dimension at most $r^2$ (with $r=rk(X)$) and $V^{(d_k)}(X)$ is its intersection with the rank-$d_k$ determinantal variety. From this structure we derive results of three kinds. \emph{Principal contributions}: (i) a matching necessary-and-sufficient \emph{discriminability theorem} showing that the minimum number of attention heads achievable by some choice of query projections is exactly $\lceil d/d_k\rceil$, providing an \emph{a priori} structural justification for the standard design $h\cdot d_k=d$ that complements the capacity-based account and addresses the gap; (ii) a tighter effective-rank version $\lceil r/d_k\rceil$ when $rk(X)=r<d$; (iii) an exact characterization $dimV(X)=r^2$ and a structural capacity bound on the family of attention operators that is independent of $N$, $d$, and $H$. Consistency checks with the established literature: we recover, within the geometric framework, the asymmetric-kernel view of , the relative-distance property of Rotary Position Embedding, and the gradient-stability advantage of Pre-LN over Post-LN. We are explicit about which results are new and which are reformulations. All theorems are accompanied by numerical experiments confirming that the predicted bounds are tight.