Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction

The corpus-mean direction of a transformer’s residual stream is a component shared acrossall inputs, and is commonly removed by mean-centering before representational analysis on theassumption that it cannot carry input-specific signal. We present evidence that it is a functionalcomponent: the reference against which the MLP gate population sets its operating point. In Phi-2, anexact decomposition of resting gate pre-activations shows that the resting inhibition of >99.9% of gatesis carried by the coupling w · bL to the carried mean direction, at 50–60× the explicit bias parameter,and the coupling is direction-specific: a random direction at matched norm orders the population’sfiring rates at ρ ≤ 0.09 (median over 50 draws; up to 0.22 in the tail) where the reference reaches0.95. (That 0.95 on its own is near-tautological, since the resting term is the mean pre-activation; wederive its null rather than quote it as evidence.) Causally, removing the stream’s projection on thereference multiplies above-threshold firing by ∼9×, dose-monotonically, at 30–40× a norm-matchedcontrol; a random-initialised twin is flat, and replacement tests show the function is carried by thedirection, not the magnitude. A direction-matched control makes the same point: noise injected alongthe reference costs 8–42× the same energy along a random direction. The decomposition replicates onfour further families spanning both gate types (GELU with an explicit gate bias, bias-free SwiGLU),and the dose-response on two of them. Across an eight-model scan the mechanism is present in everyGELU and SiLU family and absent only in OPT, where an opposing LayerNorm bias cancels thecarried reference. Gate thresholds are implemented as couplings to a constant the network builds forthe distribution it is reading (part carried in the stream and part in the parameters, in a proportion thearchitecture sets), rather than as bias parameters.