Rotation in Language Models: Simple Structure, Changes of Basis, and Conceptors

ABSTRACTThe residual stream of a transformer has no privileged coordinate system, so any description of it beginsby choosing one; factor rotation is the discipline of choosing coordinates that can be read. This reportapplies that discipline at the four places it can be applied to a language model, using GPT-2 small, and askswhat it buys for conceptors, the soft projection matrices used to read and steer a model's activations. (1)The 50,257 × 768 token lookup table, rotated to simple structure: varimax yields hundreds of nameablefactors (word families, numbers, years, suffixes, formats) and one giant factor that is the table's dominantcommon direction; promax recovers the same group factors without reducing cross-loadings, and itsfactor correlation matrix has a single large eigenvalue, the signature of a general factor. An exploratorybifactor rotation (bi-quartimin) models that structure directly: a general factor loading positively on everytoken, associated with low word frequency (r = −0.55 with Zipf frequency over 18,770 words), and groupfactors whose poles are no longer mixed with the common direction. (2) The whole model: a change ofbasis of the residual stream, orthogonal or oblique, leaves the model exactly unchanged once the weightsthat read from and write to the stream are counter-transformed and the normalization layer carries theinduced metric. The orthogonal case is known; the oblique case is a small extension with a useful reading,that the input and output embeddings then stand as pattern matrix to structure matrix. GPT-2 rewrittenin promax coordinates reproduces its own logits to 10−10; the same rewrite with ordinary LayerNorm is adifferent model. (3) Conceptors: they transform covariantly under any invertible change of basis, withgains, quota, action and Boolean-style operations invariant, but the aperture regularizer is metricdependent,so a conceptor re-fitted naively in oblique coordinates is a different filter, wrong by tens ofpercent. (4) A layer: rotating a layer's activations gives interpretable axes for that layer, and reading aconcept on them needs baselines. Against random-direction chance, a sentiment direction is absent fromthe leading directions of a general corpus until the middle of the model; against a label-shuffled null, andagainst the corpus's own sentence-final states as the reference set, the depth trend disappears and thedirection stands out at one layer only, the layer at which the sentences are also best separated. Rotationinside the concept's own subspace yields an unsupervised axis that separates held-out sentence pairs bysentiment, less sharply than the supervised difference-of-means direction does, and names the nuisancevariation beside it. Proposals for using rotation in conceptor practice follow, with the caveat that they areuntested.