Adaptive Spectral Regularization for Long-Horizon Reasoning in Autoregressive Transformers

This paper investigates why autoregressive transformers suffer from compounding reasoning errors and state-tracking failures over extended generation horizons. By modeling multi-step generation as a discrete-time dynamical system, the paper proves that unregularized singular values in attention and feed-forward projections amplify early-step representation perturbations exponentially with reasoning depth. To solve this without restricting model expressivity on deterministic tasks, the paper introduces Adaptive Spectral Regularization (ASR)—a differentiable training constraint that dynamically tightens the upper bound on projection matrix spectral norms when next-token predictive entropy is high, while relaxing the constraint during low-entropy steps. The work provides a formal upper bound on cumulative horizon drift, derives an O(dindout)O(dindout) single-step power-iteration estimator for efficient training, and establishes a controlled benchmark protocol across symbolic deduction, algorithmic state-tracking, and long-context retrieval.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24166618
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Adaptive Spectral Regularization for Long-Horizon Reasoning in Autoregressive Transformers

Akash
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
article

Adaptive Spectral Regularization for Long-Horizon Reasoning in Autoregressive Transformers

Akash
article en

Abstract

This paper investigates why autoregressive transformers suffer from compounding reasoning errors and state-tracking failures over extended generation horizons. By modeling multi-step generation as a discrete-time dynamical system, the paper proves that unregularized singular values in attention and feed-forward projections amplify early-step representation perturbations exponentially with reasoning depth. To solve this without restricting model expressivity on deterministic tasks, the paper introduces Adaptive Spectral Regularization (ASR)—a differentiable training constraint that dynamically tightens the upper bound on projection matrix spectral norms when next-token predictive entropy is high, while relaxing the constraint during low-entropy steps. The work provides a formal upper bound on cumulative horizon drift, derives an O(dindout)O(dindout) single-step power-iteration estimator for efficient training, and establishes a controlled benchmark protocol across symbolic deduction, algorithmic state-tracking, and long-context retrieval.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon