Depth-Locked Dynamics in Looped Language Models

A looped transformer applies one block of layers many times. It is trained either at a fixed number of loops r or at a number of loops drawn at random for every sequence. We measure the dynamics inside the loop of 51 looped language models trained on the same data, and find that a loss placed at one fixed depth locks the dynamics to that depth. Before r, the loop takes Euler-like steps: each step shrinks as r grows, and loops of depth 6 to 24 trace the same step profile in r steps. At r, the loop brakes: its Jacobian contracts most within one loop of r in 28 of the 31 models trained at a fixed depth through every loop, including all 27 with r ≤ 16. Past r, the step keeps rotating in a plane. At r = 6 to 12 the state swings away and back and the loss rises and falls; one turn takes about two training depths. Scaling every update by r/r′ moves this landing to loop r′ = 1.5r or 2r without retraining, and keeps the downstream accuracy of loop r. In our controls, none of these signatures appears when the depth is random, even with backpropagation through every loop, or when the loss is placed at every loop. Between r and 2r the loss of all 12 such models rises by at most 0.003 nats, against 0.02 to 0.51 nats in all 31 fixed-depth models trained through every loop. The recipes also differ in contraction, which may mediate the effect. At matched contraction, the loss of a damped fixed-depth loop still rises by 0.19 to 0.23 nats from r to 2r, that of a damped random-depth loop by at most 0.015. We prove that a loss averaged over depths confines the state in the directions the readout uses. Every registered test had its threshold written before it was run; the few that followed a first look at the data are flagged.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24201544
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Depth-Locked Dynamics in Looped Language Models

Leonardo Cofone
3 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Depth-Locked Dynamics in Looped Language Models

Leonardo Cofone
preprint en

Abstract

A looped transformer applies one block of layers many times. It is trained either at a fixed number of loops r or at a number of loops drawn at random for every sequence. We measure the dynamics inside the loop of 51 looped language models trained on the same data, and find that a loss placed at one fixed depth locks the dynamics to that depth. Before r, the loop takes Euler-like steps: each step shrinks as r grows, and loops of depth 6 to 24 trace the same step profile in r steps. At r, the loop brakes: its Jacobian contracts most within one loop of r in 28 of the 31 models trained at a fixed depth through every loop, including all 27 with r ≤ 16. Past r, the step keeps rotating in a plane. At r = 6 to 12 the state swings away and back and the loss rises and falls; one turn takes about two training depths. Scaling every update by r/r′ moves this landing to loop r′ = 1.5r or 2r without retraining, and keeps the downstream accuracy of loop r. In our controls, none of these signatures appears when the depth is random, even with backpropagation through every loop, or when the loss is placed at every loop. Between r and 2r the loss of all 12 such models rises by at most 0.003 nats, against 0.02 to 0.51 nats in all 31 fixed-depth models trained through every loop. The recipes also differ in contraction, which may mediate the effect. At matched contraction, the loss of a damped fixed-depth loop still rises by 0.19 to 0.23 nats from r to 2r, that of a damped random-depth loop by at most 0.015. We prove that a loss averaged over depths confines the state in the directions the readout uses. Every registered test had its threshold written before it was run; the few that followed a first look at the data are flagged.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon