Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction

The corpus-mean direction of a transformer’s residual stream is usually treated as a nuisance andsubtracted before analysis. We present evidence that it is a functional component: the reference againstwhich the MLP gate population sets its operating point. In Phi-2, an exact decomposition of restinggate pre-activations shows that the resting inhibition of >99.9% of gates is carried by the couplingw · bL to the carried mean direction, at 50–60× the explicit bias parameter, and this resting termrank-predicts each neuron’s firing rate at ρ ≈ 0.95. Causally, removing the stream’s projection on thereference multiplies above-threshold firing by ∼9×, dose-monotonically, at 30–40× a norm-matchedcontrol; a random-initialised twin is flat, and replacement tests show the function is carried by thedirection, not the magnitude or token content. The decomposition replicates on four further familiesspanning both gate types (GELU with an explicit gate bias, bias-free SwiGLU), and the dose-responseon two of them. Across an eight-model scan the mechanism is present in every GELU and SiLUfamily and absent only in OPT, where an opposing LayerNorm bias cancels the carried reference.Gate thresholds are implemented as couplings to a constant the network itself builds and maintains,not as bias parameters.