Semantic Phase Transitions in Transformer Observation Geometries

This technical note applies the geometric theory of semantic phase transitions in observation geometries to the internal representation spaces of transformer-based large language models (LLMs). Building on the abstract framework of observation geometries as metric–measure spaces developed in “Semantic Phase Transitions in Observation Geometries” (Takahashi, 2025, Zenodo, doi:10.5281/zenodo.17824189), it instantiates the concepts directly on layerwise hidden states of transformer language models. Each transformer layer is treated as an observation geometry obtained from hidden representations under a natural Euclidean metric and an empirical measure induced by a prompt distribution. On top of these spaces, the note defines two families of graphs: distance-based random geometric graphs and attention-based random connection models that use multi-head self-attention weights as internal connectivity scores. Semantic phase transitions are defined as qualitative changes in large-scale connectivity (in particular, the emergence or disappearance of a giant component containing task-relevant nodes) along natural parameter families such as depth, model size, training step, or decoding temperature. The note also provides an explicit, implementation-oriented protocol for detecting semantic phase transitions in practice, with a focus on in-context learning tasks where performance is known to show sharp, scale-dependent changes. The protocol describes how to construct layerwise graphs on query-token representations, label nodes by task correctness, and track graph observables (e.g. giant-component fraction) alongside task accuracy across model families or checkpoints. Rather than introducing new asymptotic percolation theorems, this work serves as a bridge between the general theory of semantic phase transitions on observation spaces and concrete transformer architectures. It offers a geometrically principled way to link emergent behaviours in LLMs to changes in representation-space connectivity, and provides a practical template for empirical studies on current large language models.