Adam Optimizer at the Edge of Stability: Exact Thresholds for Period-Two Oscillations
On f(w) = (λ/2)w², Adam without bias correction has an exact period-two orbit wt = (−1)tA with A = η(1−β₁)/(2(1+β₁)) − ε/λ. On it the second moment equals gt², and the preconditioned curvature equals the adaptive edge-of-stability threshold 2(1+β₁)/(η(1−β₁)) of Cohen et al. Bock and Weiß (2019) found this orbit at ε = 0. They observed that its multipliers depend on (β₁, β₂) only, and their plots show it unstable for β₁, β₂ near 1. They obtained lengthy symbolic eigenvalue expressions but left an analytic description of the stability region unresolved. Fong and Yang (2026) give the orbit for ε > 0. We prove that the Floquet multipliers are the squares of the roots of an explicit cubic in (β₁, β₂, θ), where θ = A/A⋆ ∈ (0, 1] measures ε. Apart from the degenerate sign-descent corner, the orbit is linearly asymptotically stable if and only if (1−β₁)(1−β₁β₂) > 2θβ₁(1+β₂), with an explicit unit-circle factorization at the boundary. At ε = 0 it is stable for every β₂ ∈ [0, 1) when 0 < β₁ ≤ 3 − 2√2 ≈ 0.172, and strictly linearly unstable for every β₂ when β₁ > 1/3. At (0.9, 0.999) the dominant linear growth factor is 1.0494 per step. At these defaults, a positive ε stabilises the orbit only when λA⋆ exceeds ε by less than 0.28%. The same cubic, with a gradient elasticity κ in place of 1, governs the symmetric orbit of every even loss. A unified, division-free criterion gives its full stability interval, including both real and oscillatory boundaries. At ε = 0 this reduces to 0 < κ < κ⋆ outside the degenerate corner; κ⋆ = 0.9906 at the defaults, so the quadratic lies just on the unstable side. On quadratics in ℝd, every sign cycle of coordinate-wise Adam is unstable at the defaults, and these cycles persist, still unstable, near every non-degenerate minimum of a C3 loss for small step size. A decoupled weight decay stabilises the orbit at the defaults only if it removes more than 97% of the weight per step. At ε = 0 the loss of stability is a sub-critical Neimark–Sacker bifurcation for every β₂ ≠ 2 − √3: we give the first Lyapunov coefficient in closed form and prove it positive. For ε > 0 the criticality can change; along β₁ = 0.9 it does so exactly once, at β₂ = 0.98948. In one dimension the two-cycle and its Floquet cubic are exact for losses without symmetry. Simulations at the defaults exhibit sustained irregular motion, with a finite-time largest Lyapunov exponent of about 5 × 10−3 per step. On these trajectories the preconditioned curvature averages 0.93 of the threshold and exceeds it on 18.5% of the steps, and the loss averages 1.17 times the loss on the orbit. Below the stability boundary, distinct initializations provide numerical evidence of coexistence with a large-amplitude regime. Dedication. The mathematical analysis developed in this work is dedicated with unconditional love and gratitude to Giovanna Elizabeth Pereira de Matos Costa, the author’s mother. Throughout my entire life, on the bench and at home, you taught by example that true judgment never rests on appearances or conventional defaults, but looks past the surface to uncover hidden flaws with rigor, fairness, and unwavering integrity. Just as this paper exposes the instability beneath an assumed equilibrium, may these theorems stand as permanent tribute to the judge who taught me to seek the exact truth—and to the mother whose love remains the one true constant in my life.
Authors
- Gabriel Paiva (ORCID: https://orcid.org/0009-0007-8382-3238)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-15
- DOI
- https://doi.org/10.5281/zenodo.24239329
- Primary Topic
- Stochastic Gradient Optimization Techniques
- Type
- preprint