Class Separation Dynamics in Vision Transformers: An Empirical Study

Vision Transformers (ViTs) have emerged in recent years as a powerful architecture for image classification tasks and beyond. However, due to their back-box nature, our understanding of their learning dynamics is minimal. In this work, we study ViT learning dynamics through the lens of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">class separation</i>, tracking inter- and intra-class structure, propagating from the patch-embedding layer to the classifier head. Across four datasets and extensive hyper-parameter configurations, we characterize two primary regularities. Firstly, we observe a layer-wise consistency, where separation fuzziness spikes between the input projection and the first transformer block, then decays exponentially with depth. Secondly, provided that the first law holds, fuzziness at the final layer also decays exponentially as a function of the training epochs. The persistence of these laws across datasets and hyper-parameters suggests an intrinsic relationship to ViT training. Our study supplies a quantitative foundation for interpreting the design of transformer-based vision models. Code and reproducibility scripts are available through https://github.com/DaraVaram/vit-data-separation.