Why Gradient Descent Stalls on Raw Data: Condition Numbers of Six UCI Regression Designs, Exact Step-Size and Heavy-Ball Rates, and an Intercept Identity That Explains Condition Numbers Above Ten Billion

Gradient descent on a least-squares loss converges at a speed fixed by one number, the condition number of the Hessian. I compute that number for six public UCI regression data sets (combined cycle power plant, concrete, airfoil noise, building energy, white wine and yacht hydrodynamics), each with an intercept, using raw, centred and standardised features. Raw designs have condition numbers between 630,000 and 270 billion (median 35 billion). Standardising brings them down to between 12 and 831 (median 116). Centring alone is not enough: it leaves 5 of 6 designs above 10,000. I prove that for an intercept plus one feature with mean m and standard deviation s, the condition number kappa satisfies kappa + 2 + 1/kappa = (1 + m^2 + s^2)^2 / s^2 exactly, so one feature such as atmospheric pressure (mean 1013, sd 5.9) makes kappa about 30 billion. I also prove the step-size stability limit 2/L, the best fixed step 2/(L + mu) with rate (kappa - 1)/(kappa + 1), and that Polyak's heavy ball with tuned parameters has every mode contracting at exactly (sqrt(kappa) - 1)/(sqrt(kappa) + 1). On the standardised designs, simulated losses match the exact spectral formula to within 5e-12 of the initial loss; steps of 1.99/L converge and 2.01/L diverge on every data set; and heavy ball reaches a one-millionth loss reduction in 12 to 167 iterations, against 24 to 2,341 for gradient descent with step 1/L. On the raw designs, the theory says gradient descent with step 1/L would need between 830,000 and 610 billion iterations.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-06
DOI
https://doi.org/10.5281/zenodo.23992204
Primary Topic
Stochastic Gradient Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Why Gradient Descent Stalls on Raw Data: Condition Numbers of Six UCI Regression Designs, Exact Step-Size and Heavy-Ball Rates, and an Intercept Identity That Explains Condition Numbers Above Ten Billion

Manoj Kumar Aguru
4 links
Zenodo (CERN European Organization for Nuclear Research)
Stochastic Gradient Optimization Techniques
preprint

Why Gradient Descent Stalls on Raw Data: Condition Numbers of Six UCI Regression Designs, Exact Step-Size and Heavy-Ball Rates, and an Intercept Identity That Explains Condition Numbers Above Ten Billion

Manoj Kumar Aguru
preprint en

Abstract

Gradient descent on a least-squares loss converges at a speed fixed by one number, the condition number of the Hessian. I compute that number for six public UCI regression data sets (combined cycle power plant, concrete, airfoil noise, building energy, white wine and yacht hydrodynamics), each with an intercept, using raw, centred and standardised features. Raw designs have condition numbers between 630,000 and 270 billion (median 35 billion). Standardising brings them down to between 12 and 831 (median 116). Centring alone is not enough: it leaves 5 of 6 designs above 10,000. I prove that for an intercept plus one feature with mean m and standard deviation s, the condition number kappa satisfies kappa + 2 + 1/kappa = (1 + m^2 + s^2)^2 / s^2 exactly, so one feature such as atmospheric pressure (mean 1013, sd 5.9) makes kappa about 30 billion. I also prove the step-size stability limit 2/L, the best fixed step 2/(L + mu) with rate (kappa - 1)/(kappa + 1), and that Polyak's heavy ball with tuned parameters has every mode contracting at exactly (sqrt(kappa) - 1)/(sqrt(kappa) + 1). On the standardised designs, simulated losses match the exact spectral formula to within 5e-12 of the initial loss; steps of 1.99/L converge and 2.01/L diverge on every data set; and heavy ball reaches a one-millionth loss reduction in 12 to 167 iterations, against 24 to 2,341 for gradient descent with step 1/L. On the raw designs, the theory says gradient descent with step 1/L would need between 830,000 and 610 billion iterations.

Zenodo (CERN European Organization for Nuclear Research)
Chaitanya (IN)
Stochastic Gradient Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Why Gradient Descent Stalls on Raw Data: Condition Numbers of Six UCI Regression Designs, Exact Step-Size and Heavy-Ball Rates, and an Intercept Identity That Explains Condition Numbers Above Ten Billion — Manoj Kumar Aguru · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS