Gradient flow for deep factored scalar linear attention learns multi-step gradient descent

We study in-context learning of scalar linear regression by L untied linear-attention layers, each with a trainable key scalar k_i and query scalar q_i. For every depth L and prompt length n, population gradient flow on the 2L factors is bounded and converges to a critical point, every critical point that is not a global minimizer has a negative Hessian eigenvalue, and the flow reaches a global minimizer from all initializations outside a Lebesgue null set, for instance almost surely under Gaussian initialization. The trained network performs L gradient descent steps on the prompt's least-squares loss, with step sizes k_iq_i the reciprocals of the zeros of a rescaled Laguerre polynomial L_L^(n/2), and its risk is $2^LL!/\prod_{j=1}^{L}(n+2j)$. Initializations with k_i = -q_i in some layer converge to a critical point that is not a minimizer, so the null set is not empty.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24245275
Primary Topic
Stochastic Gradient Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Gradient flow for deep factored scalar linear attention learns multi-step gradient descent

Yitzchak Shmalo
Zenodo (CERN European Organization for Nuclear Research)
Stochastic Gradient Optimization Techniques
preprint

Gradient flow for deep factored scalar linear attention learns multi-step gradient descent

Yitzchak Shmalo
preprint en

Abstract

We study in-context learning of scalar linear regression by L untied linear-attention layers, each with a trainable key scalar k_i and query scalar q_i. For every depth L and prompt length n, population gradient flow on the 2L factors is bounded and converges to a critical point, every critical point that is not a global minimizer has a negative Hessian eigenvalue, and the flow reaches a global minimizer from all initializations outside a Lebesgue null set, for instance almost surely under Gaussian initialization. The trained network performs L gradient descent steps on the prompt's least-squares loss, with step sizes k_iq_i the reciprocals of the zeros of a rescaled Laguerre polynomial L_L^(n/2), and its risk is $2^LL!/\prod_{j=1}^{L}(n+2j)$. Initializations with k_i = -q_i in some layer converge to a critical point that is not a minimizer, so the null set is not empty.

Zenodo (CERN European Organization for Nuclear Research)
Hebrew University of Jerusalem (IL)
Stochastic Gradient Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon