Normalized State Decomposition: The Hidden Geometry of Neural Network Representations

For decades, normalization (LayerNorm, BatchNorm, RMSNorm) has been widely regarded as an engineering technique for stabilizing deep neural network training, while its mathematical essence has remained unelucidated. Meanwhile, the field of neural network interpretability has long been trapped in a black-box dilemma---SAEs require training additional models and produce unstable features, while mechanistic interpretability struggles to scale. To address these challenges, this paper presents a unified mathematical framework showing that the core formula s_d = ± |v_d| / ||v|| serves as the mathematical cornerstone of neural network normalization, providing a complete and deterministic explanation of how neural networks encode and process information. Normalized networks decompose representations into normalized direction vectors on a hypersphere (termed semantic vectors s = v/||v||, capturing pure directional information) and a stability/confidence scalar ||v||. It should be noted that the term "semantic" in this paper refers to the intrinsic high-dimensional geometric directionality of neural networks, rather than next-generation linguistic semantics (formally defined in Section 4). Neural networks then operate as high-dimensional navigators: the LM Head weights define a fixed coordinate system on the hypersphere, and forward propagation traces trajectories in this space, with each of the D hidden dimensions serving as a control axis that pushes and pulls the probability mass of all V vocabulary tokens. Neuron specialization operates at two levels---symbolic format detection (numbers, code, punctuation) and co-occurrence pattern detection (e.g., "[animal] + is + a" syntactic structure)---revealing that seemingly "concept recognition" is actually format pattern matching. This framework provides the mathematical foundation for Flash Interpretability (Davies, 2025) and is universally applicable to all architectures (Transformer, MoE, ViT, autoencoders, Neural ODEs), resolving core interpretability puzzles: polysemanticity arises from shared format detection, superposition from limited dimensional capacity, and the linear representation hypothesis from hyperspherical geometry. Experimental evidence demonstrating that SAEs are unnecessary is provided in Section 11.