Orientation Over Capacity: A Stabiliser-Theoretic Analysis of Representational Vulnerability in Neural Networks

v2 — Corrected formal revision This version corrects the mathematical formulation of representational blind sets, layerwise invariance, distribution-dependent sensitivity, and sensitivity selectivity. The central empirical results and the main orientation-dependent interpretation remain unchanged. -- Abstract Neural networks with identical architecture and dimension can exhibit different sensitivity to the same family of input perturbations. We formalise this observation through the symmetry-matching condition Stab(G,F) ∩ E= {e}, adapted from prior algebraic work on finite-alphabet error detection, and extend it to continuous representations by defining a representation-level blind set Srep (the set of perturbations that leave the encoding exactly invariant) and three distinct scalars, namely the exact representational vulnerability Vrep ∈ [0,1] (the fraction of a perturbation family in the blind set), its tolerance relaxation Vε (the fraction of near -invisible perturbations), and a selectivity index Gsel (the Gini coefficient of the sensitivity distribution). Gsel quantifies inequality in perturbation sensitivity, while exact invariance is described by Vrep and Vε. We prove that Srep ⊆ Sout holds for any layered architecture, and that the exact blind set is non-decreasing with depth (formally, Srep(ϕl) ⊆ Srep(ϕl+1) for any deterministic layer, with equality at injective layers), so absorbed information cannot be recovered downstream. We further establish, by an explicit example and a minimum-gain margin γE (W), that vulnerability depends on weight-matrix orientation at fixed dimension. On an Elhage-style toy model of superposition (20 features, d∈ {2,...,20}, sparsity 0.05) the encoding sensitivity to an additive coordinate perturbation reduces analytically to the column norm ∥Wei∥, so the toy study measures the selectivity Gsel of these column norms; we find (i) Gsel decreases with hidden dimension, (ii) at fixed d= 5, Gsel varies substantially across 15 random initialisations under both the standard and equal-importance settings, with ranges of approximately 3.1× and 1.7×, respectively, and (iii) selecting the best of 15 standard initialisations at d= 5 reaches a lower Gsel than the mean of five d=10 initialisations, an initialisation-variance finding rather than a trained orientation objective. On two pretrained transformer language models (Qwen 2.5-0.5B, SmolLM2-360M) we show that (iv) per-layer mean local sensitivity is negatively associated with depth while Gsel is positively associated with depth, indicating growing selectivity, and (v) six structured perturbation families form a sensitivity hierarchy in which numeric-value changes produce local responses comparable to synonym substitutions, a local pattern subject to tokenisation confounds we make explicit, motivating downstream and tokenisation-controlled evaluation of its functional consequences.