WEDGE: Weight-Equivariant Dikin Gauss–Newton, Dual Diagonal Geometries, and O(K⁻²) Step Certificates
We study open questions posed by Liu et al. (arXiv:2510.13680) ("Adam or Gauss–Newton?") and give partial answers; we also report an issue in the released layerwise code of Abreu et al. (arXiv:2510.09378) ("Full Gauss–Newton"). The results of version 1 of this note are kept. (i) For the full-batch comparison of the diagonal preconditioners diag(Σ)^(−1) and diag(Σ)^(−1/2) (their Appendix A.3), we separate the readings diag(Σ)^p and diag(Σ^p), show that their condition-number ratios have the same supremum over Σ in every dimension, and bound it by d − 2√d + 9/4. The criterion for a ratio above one is the classical Forsythe–Straus/Shapiro optimal-scaling condition; the supremum is 3/2, 2, 5/2 for d = 3, 4, 5 (matching Shapiro's 1991 lower bound d/2), lies in [3.0224, 73/24] for d = 6 (we believe the lower value is exact), and equals d − 2√d + 2 + O(d^(−1/2)). (ii) For the stochastic regime of their Lemma 2 we prove κ_s(A(P)) ≥ 2/λ_min(C) + d − 2 for every diagonal P, so the guarantee of GN^(−1/2) is at most d/2 times better than that of GN^(−1), and this is sharp; the same bound limits what importance sampling can gain over uniform sampling in the one-step guarantee of randomized Kaczmarz on square nonsingular systems; on every consistent system with n rows and rank at least 2 the gain is at most n/2, and this is sharp. For the exact rate of the expected loss, we prove by hand that the supremum of the same ratio lies between d/2 and d/2 + 11/2 for every d ≥ 3, so it is d/2 + O(1); it is at most d/2 + 3/2 for every d ≥ 3 (by hand for d ≥ 31) and at most d/2 + (√6−1)²/2 + o(1) as d → ∞, at most d/2 + 0.052 for 3 ≤ d ≤ 30, and for d = 3 it equals 3/2 and is not attained (both computer-assisted). We conjectured d/2 for every d ≥ 4; a companion paper proves this, with computer assistance. (iii) We extend their Lemma 1 to batch size B with label noise, to non-Gaussian kurtosis, and to softmax cross-entropy under uniform calibration. (iv) On their squared-logit model, a coordinatewise gradient-dependent ridge cancels the frequency weights, and its no-overshoot threshold is 1/θ* on their label range (proved; one step is an exact polynomial certificate in rational arithmetic). (v) A pre-registered test of their Section 7 hypothesis on two language models of up to 162M parameters is consistent with it in transformer-block weights late in training; several of our predictions failed, and we report them. (vi) In the layerwise code of Abreu et al. (arXiv:2510.09378), a prefix-matching issue makes one layer's subproblem include two others in 12-block models; in a small replication the one-character fix changes the final loss by +0.0017 nats. From version 1 we keep a classical (Kalman) step-size recursion with a smooth resolvent cooling weight, a weight-equivariant damping rule with a gradient-power trichotomy, Taylor-jet step certificates that bound the loss along one direction, and controlled GPU evaluations. A compressed-gate gain on modded-nanogpt Track 3 reported in version 1 was reversed in a single-seed full-horizon run and is now in an appendix as a preliminary observation. Version 4 (17 November 2026). Changes with respect to version 3 are listed in the appendix "Changes". Companion papers: "On two questions of Shapiro on Jacobi scaling", doi:10.5281/zenodo.24315181, and "A sharp factor d/2 for diagonal preconditioning of single-sample SGD", doi:10.5281/zenodo.24315183. Files: the paper (PDF) and the verification scripts (supplement_1_main.zip); the data archives supplement_2 and supplement_3 are unchanged from version 3.
Authors
- Gabriel Paiva (ORCID: https://orcid.org/0009-0007-8382-3238)
Institutions
- Elizabethtown Community Hospital (US)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24291386
- Citations
- 1
- Primary Topic
- Stochastic Gradient Optimization Techniques
- Type
- preprint