WEDGE: Weight-Equivariant Dikin Gauss–Newton, Dual Diagonal Geometries, and O(K⁻²) Step Certificates

We study open questions posed by Liu et al. (arXiv:2510.13680) ("Adam or Gauss–Newton?") and give partial answers; we also report an issue in the released layerwise code of Abreu et al. (arXiv:2510.09378) ("Full Gauss–Newton"). The results of version 1 of this note are kept. (i) For the full-batch comparison of the diagonal preconditioners diag(Σ)^(−1) and diag(Σ)^(−1/2) (their Appendix A.3), we separate the readings diag(Σ)^p and diag(Σ^p), show that their condition-number ratios have the same supremum over Σ in every dimension, and bound it by d − 2√d + 9/4. The criterion for a ratio above one is the classical Forsythe–Straus/Shapiro optimal-scaling condition; the supremum is 3/2, 2, 5/2 for d = 3, 4, 5 (matching Shapiro's 1991 lower bound d/2), lies in [3.0224, 73/24] for d = 6 (we believe the lower value is exact), and equals d − 2√d + 2 + O(d^(−1/2)). (ii) For the stochastic regime of their Lemma 2 we prove κ_s(A(P)) ≥ 2/λ_min(C) + d − 2 for every diagonal P, so the guarantee of GN^(−1/2) is at most d/2 times better than that of GN^(−1), and this is sharp; the same bound limits what importance sampling can gain over uniform sampling in the one-step guarantee of randomized Kaczmarz on square nonsingular systems; on every consistent system with n rows and rank at least 2 the gain is at most n/2, and this is sharp. For the exact rate of the expected loss, we prove by hand that the supremum of the same ratio lies between d/2 and d/2 + 11/2 for every d ≥ 3, so it is d/2 + O(1); it is at most d/2 + 3/2 for every d ≥ 3 (by hand for d ≥ 31) and at most d/2 + (√6−1)²/2 + o(1) as d → ∞, at most d/2 + 0.052 for 3 ≤ d ≤ 30, and for d = 3 it equals 3/2 and is not attained (both computer-assisted). We conjectured d/2 for every d ≥ 4; a companion paper proves this, with computer assistance. (iii) We extend their Lemma 1 to batch size B with label noise, to non-Gaussian kurtosis, and to softmax cross-entropy under uniform calibration. (iv) On their squared-logit model, a coordinatewise gradient-dependent ridge cancels the frequency weights, and its no-overshoot threshold is 1/θ* on their label range (proved; one step is an exact polynomial certificate in rational arithmetic). (v) A pre-registered test of their Section 7 hypothesis on two language models of up to 162M parameters is consistent with it in transformer-block weights late in training; several of our predictions failed, and we report them. (vi) In the layerwise code of Abreu et al. (arXiv:2510.09378), a prefix-matching issue makes one layer's subproblem include two others in 12-block models; in a small replication the one-character fix changes the final loss by +0.0017 nats. From version 1 we keep a classical (Kalman) step-size recursion with a smooth resolvent cooling weight, a weight-equivariant damping rule with a gradient-power trichotomy, Taylor-jet step certificates that bound the loss along one direction, and controlled GPU evaluations. A compressed-gate gain on modded-nanogpt Track 3 reported in version 1 was reversed in a single-seed full-horizon run and is now in an appendix as a preliminary observation. Version 4 (17 November 2026). Changes with respect to version 3 are listed in the appendix "Changes". Companion papers: "On two questions of Shapiro on Jacobi scaling", doi:10.5281/zenodo.24315181, and "A sharp factor d/2 for diagonal preconditioning of single-sample SGD", doi:10.5281/zenodo.24315183. Files: the paper (PDF) and the verification scripts (supplement_1_main.zip); the data archives supplement_2 and supplement_3 are unchanged from version 3.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24291386
Citations
1
Primary Topic
Stochastic Gradient Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

WEDGE: Weight-Equivariant Dikin Gauss–Newton, Dual Diagonal Geometries, and O(K⁻²) Step Certificates

Gabriel Paiva
1 citations
Zenodo (CERN European Organization for Nuclear Research)
Stochastic Gradient Optimization Techniques
preprint

WEDGE: Weight-Equivariant Dikin Gauss–Newton, Dual Diagonal Geometries, and O(K⁻²) Step Certificates

Gabriel Paiva
preprint en
1 citations

Abstract

We study open questions posed by Liu et al. (arXiv:2510.13680) ("Adam or Gauss–Newton?") and give partial answers; we also report an issue in the released layerwise code of Abreu et al. (arXiv:2510.09378) ("Full Gauss–Newton"). The results of version 1 of this note are kept. (i) For the full-batch comparison of the diagonal preconditioners diag(Σ)^(−1) and diag(Σ)^(−1/2) (their Appendix A.3), we separate the readings diag(Σ)^p and diag(Σ^p), show that their condition-number ratios have the same supremum over Σ in every dimension, and bound it by d − 2√d + 9/4. The criterion for a ratio above one is the classical Forsythe–Straus/Shapiro optimal-scaling condition; the supremum is 3/2, 2, 5/2 for d = 3, 4, 5 (matching Shapiro's 1991 lower bound d/2), lies in [3.0224, 73/24] for d = 6 (we believe the lower value is exact), and equals d − 2√d + 2 + O(d^(−1/2)). (ii) For the stochastic regime of their Lemma 2 we prove κ_s(A(P)) ≥ 2/λ_min(C) + d − 2 for every diagonal P, so the guarantee of GN^(−1/2) is at most d/2 times better than that of GN^(−1), and this is sharp; the same bound limits what importance sampling can gain over uniform sampling in the one-step guarantee of randomized Kaczmarz on square nonsingular systems; on every consistent system with n rows and rank at least 2 the gain is at most n/2, and this is sharp. For the exact rate of the expected loss, we prove by hand that the supremum of the same ratio lies between d/2 and d/2 + 11/2 for every d ≥ 3, so it is d/2 + O(1); it is at most d/2 + 3/2 for every d ≥ 3 (by hand for d ≥ 31) and at most d/2 + (√6−1)²/2 + o(1) as d → ∞, at most d/2 + 0.052 for 3 ≤ d ≤ 30, and for d = 3 it equals 3/2 and is not attained (both computer-assisted). We conjectured d/2 for every d ≥ 4; a companion paper proves this, with computer assistance. (iii) We extend their Lemma 1 to batch size B with label noise, to non-Gaussian kurtosis, and to softmax cross-entropy under uniform calibration. (iv) On their squared-logit model, a coordinatewise gradient-dependent ridge cancels the frequency weights, and its no-overshoot threshold is 1/θ* on their label range (proved; one step is an exact polynomial certificate in rational arithmetic). (v) A pre-registered test of their Section 7 hypothesis on two language models of up to 162M parameters is consistent with it in transformer-block weights late in training; several of our predictions failed, and we report them. (vi) In the layerwise code of Abreu et al. (arXiv:2510.09378), a prefix-matching issue makes one layer's subproblem include two others in 12-block models; in a small replication the one-character fix changes the final loss by +0.0017 nats. From version 1 we keep a classical (Kalman) step-size recursion with a smooth resolvent cooling weight, a weight-equivariant damping rule with a gradient-power trichotomy, Taylor-jet step certificates that bound the loss along one direction, and controlled GPU evaluations. A compressed-gate gain on modded-nanogpt Track 3 reported in version 1 was reversed in a single-seed full-horizon run and is now in an appendix as a preliminary observation. Version 4 (17 November 2026). Changes with respect to version 3 are listed in the appendix "Changes". Companion papers: "On two questions of Shapiro on Jacobi scaling", doi:10.5281/zenodo.24315181, and "A sharp factor d/2 for diagonal preconditioning of single-sample SGD", doi:10.5281/zenodo.24315183. Files: the paper (PDF) and the verification scripts (supplement_1_main.zip); the data archives supplement_2 and supplement_3 are unchanged from version 3.

Zenodo (CERN European Organization for Nuclear Research)
Elizabethtown Community Hospital (US)
Stochastic Gradient Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon