Why Gradient Descent Stalls on Raw Data: Condition Numbers of Six UCI Regression Designs, Exact Step-Size and Heavy-Ball Rates, and an Intercept Identity That Explains Condition Numbers Above Ten Billion
Gradient descent on a least-squares loss converges at a speed fixed by one number, the condition number of the Hessian. I compute that number for six public UCI regression data sets (combined cycle power plant, concrete, airfoil noise, building energy, white wine and yacht hydrodynamics), each with an intercept, using raw, centred and standardised features. Raw designs have condition numbers between 630,000 and 270 billion (median 35 billion). Standardising brings them down to between 12 and 831 (median 116). Centring alone is not enough: it leaves 5 of 6 designs above 10,000. I prove that for an intercept plus one feature with mean m and standard deviation s, the condition number kappa satisfies kappa + 2 + 1/kappa = (1 + m^2 + s^2)^2 / s^2 exactly, so one feature such as atmospheric pressure (mean 1013, sd 5.9) makes kappa about 30 billion. I also prove the step-size stability limit 2/L, the best fixed step 2/(L + mu) with rate (kappa - 1)/(kappa + 1), and that Polyak's heavy ball with tuned parameters has every mode contracting at exactly (sqrt(kappa) - 1)/(sqrt(kappa) + 1). On the standardised designs, simulated losses match the exact spectral formula to within 5e-12 of the initial loss; steps of 1.99/L converge and 2.01/L diverge on every data set; and heavy ball reaches a one-millionth loss reduction in 12 to 167 iterations, against 24 to 2,341 for gradient descent with step 1/L. On the raw designs, the theory says gradient descent with step 1/L would need between 830,000 and 610 billion iterations.
Authors
- Manoj Kumar Aguru
Institutions
- Chaitanya (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-06
- DOI
- https://doi.org/10.5281/zenodo.23992204
- Primary Topic
- Stochastic Gradient Optimization Techniques
- Type
- preprint