Gradient flow for deep factored scalar linear attention learns multi-step gradient descent
We study in-context learning of scalar linear regression by L untied linear-attention layers, each with a trainable key scalar k_i and query scalar q_i. For every depth L and prompt length n, population gradient flow on the 2L factors is bounded and converges to a critical point, every critical point that is not a global minimizer has a negative Hessian eigenvalue, and the flow reaches a global minimizer from all initializations outside a Lebesgue null set, for instance almost surely under Gaussian initialization. The trained network performs L gradient descent steps on the prompt's least-squares loss, with step sizes k_iq_i the reciprocals of the zeros of a rescaled Laguerre polynomial L_L^(n/2), and its risk is $2^LL!/\prod_{j=1}^{L}(n+2j)$. Initializations with k_i = -q_i in some layer converge to a critical point that is not a minimizer, so the null set is not empty.
Authors
- Yitzchak Shmalo (ORCID: https://orcid.org/0000-0002-2886-3049)
Institutions
- Hebrew University of Jerusalem (IL)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-15
- DOI
- https://doi.org/10.5281/zenodo.24245275
- Primary Topic
- Stochastic Gradient Optimization Techniques
- Type
- preprint