What are activation plateaus?

This working paper summarizes my current understanding of the "activation plateau" phenomenon. The text is written without significant AI assistance (only proofreading). Almost all experiments are based on past publications, pre-prints, or blog posts, but reimplemented using generative AI. Abstract: I am proposing a new angle to approach fundamental mechanistic interpretability, based on looking for error correction and privileged features in neural network activations. Collaborators and I have explored a few directions in this area over the last two years, but I now think it is promising enough to focus on full-time. In this document I summarize the surprising activation-space phenomenology we discovered, and discuss open questions and future work. Our primary observation is that the activation space of LLMs (and other deep neural networks) contains stable regions (“activation plateaus”) within which the model’s output is nearly constant. Within these regions LLMs are robust to activation perturbations, and moving outside a region causes an abrupt jump in later activations and model output. All model inputs map to activations within stable regions, but almost all random locations in activation space do not exhibit this robustness property. The existence and sharpness of activation plateaus are surprising because the common continuous linear representation assumption would predict smooth output changes in response to smooth activation changes. The observation might indicate a kind of error correction mechanism as suggested in the superposition literature, and hence may provide insight into what the computational features of a model are. We have some evidence that feature-like directions are empirically privileged by activation plateau geometry which could allow us to identify computational features directly from the model. I am also very interested in how the locations of all activation plateaus are encoded in models, whether they are isolated regions or form a continuous manifold, and what we can learn about LLM representations. I believe activation plateaus are an important part of how LLMs work. The phenomenon emerges with scale—it strengthens with larger model size and more training data—and does not occur in random Transformers. It is an emergent property of the MLP blocks, independent of attention and LayerNorm. Activation plateaus broadly reproduce in every sufficiently large LLM we tested, and appear in deep image models (Inception models and vision transformers) as well.