Averaged Operators and the Slow Flow of Depth in Trained Transformers

The depth of a trained transformer is studied as a discrete dynamical system: each layer advances a hidden state, and the network’s prediction is the endpoint of the evolution. We show empirically that this evolution is well described by a slowly varying effective field. A band of consecutive layers is reproduced as an input–output map by repeated application of a single averaged operator, defined as the mean of the layers’ update fields, with every weight taken from the pretrained model and nothing fitted; a forty-eight-layer GPT-2 network is reproduced by as few as four to eight such operators. The mechanism is a separation of scales: each layer’s update is the sum of a slowly varying component shared by neighboring layers and a fast component that is nearly orthogonal from layer to layer, and averaging suppresses the fast component according to a quantitative law given in the text while preserving the slow one. The admissible averaging width is small in the first layers and grows with depth; there is a sharp obstruction in the first layers caused by a sign reversal of the layer fields. Resolving the slow drift to first and second order, as in higher-order finite elements, reduces the error and halves the number of operators required. The averaged operator is more computationally expensive to evaluate than the layers it replaces, so its purpose is analysis of the computation rather than compression. The construction reproduces across GPT-2 medium, large, and XL, with a preliminary check indicating partial transfer to OPT.

Authors

Institutions

Publication Details

Journal
Computation
Published
2026-11-13
DOI
https://doi.org/10.3390/computation14110289
Primary Topic
Neural Networks and Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Averaged Operators and the Slow Flow of Depth in Trained Transformers

Vitaly A Volpert
1 links
Computation
Neural Networks and Applications
article

Averaged Operators and the Slow Flow of Depth in Trained Transformers

Vitaly A Volpert
article en

Abstract

The depth of a trained transformer is studied as a discrete dynamical system: each layer advances a hidden state, and the network’s prediction is the endpoint of the evolution. We show empirically that this evolution is well described by a slowly varying effective field. A band of consecutive layers is reproduced as an input–output map by repeated application of a single averaged operator, defined as the mean of the layers’ update fields, with every weight taken from the pretrained model and nothing fitted; a forty-eight-layer GPT-2 network is reproduced by as few as four to eight such operators. The mechanism is a separation of scales: each layer’s update is the sum of a slowly varying component shared by neighboring layers and a fast component that is nearly orthogonal from layer to layer, and averaging suppresses the fast component according to a quantitative law given in the text while preserving the slow one. The admissible averaging width is small in the first layers and grows with depth; there is a sharp obstruction in the first layers caused by a sign reversal of the layer fields. Resolving the slow drift to first and second order, as in higher-order finite elements, reduces the error and halves the number of operators required. The averaged operator is more computationally expensive to evaluate than the layers it replaces, so its purpose is analysis of the computation rather than compression. The construction reproduces across GPT-2 medium, large, and XL, with a preliminary check indicating partial transfer to OPT.

ComputationVol. 14(11)
Université Claude Bernard Lyon 1 (FR), Centre National de la Recherche Scientifique (FR), Institut Camille Jordan (FR), Institute of Fundamental Technological Research (PL), Polish Academy of Sciences (PL)
Neural Networks and Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon