OFF-MANIFOLD BY CONSTRUCTION: INTERMEDIATE-LAYER ADAPTERS IN FROZEN AR DECODERS

A common approach to adapting frozen autoregressive transformers without modifying their weights is to perturb hidden states at intermediate layers, for example, via element-wise modulation or residual bottlenecks. We prove that any such perturbation, provided it is real-analytic and non-degenerate in its parameters, produces hidden states that, with probability 1 over initialization, lie outside the model’s natural reachable set. Because subsequent layers are real-analytic maps, this off-manifold deviation propagates forward rather than being absorbed, which shifts the output token distribution away from the natural distribution. In cascaded architectures with downstream decoders, the perturbed hidden states further break the coupled training graph, causing distribution shift in downstream modules. We verify the analyticity assumptions for standard transformer components (SiLU, SwiGLU, RMSNorm, softmax) and characterize the empirical behavior of such adapters on Qwen3-TTS 1.7B with EiLM. The results establish that intermediate-layer hidden-state perturbation of frozen AR decoders is almost surely off-manifold at initialization and that the resulting deviation is empirically non-negligible.