How Far is Adam from Natural Gradient Descent?

Adam is the default optimizer for modern deep learning, yet its geometric relationship to natural gradient descent (NGD) remains incompletely characterized. Adam’s update rule implicitly approximates the true Fisher information matrix through three successive simplifications, namely diagonal truncation, empirical substitution of observed for model predicted labels, and temporal lag introduced by the exponential moving average. While Hwang (2024) formalizes Adam’s connection to NGD under simplifying assumptions (β1 = 0, bounded gradients), the geometric cost of Adam’s full update rule with momentum remains unquantified. We apply the Wu et al. (2024) γ(∆θ) evaluation framework directly to Adam across four controlled loss landscape regimes spanning convex, illconditioned, classification, and non-convex settings, measuring Adam’s geometric deviation from true NGD at each training step. Our experiments reveal that Adam’s γ(∆θ) is context-dependent rather than uniformly poor, falling below SGD and iEF in the well-conditioned regime, rising above both under ill-conditioning and misspecification, and reaching ≈ 1000 in the non-convex neural network regime. γ and training loss appear inversely related for Adam across regimes, with Adam achieving a low loss floor in every experiment despite its severe geometric deviation from the natural gradient. We additionally find that iEF produces meaningfully more stable γ trajectories than EF, which oscillates and fails to converge, suggesting that geometric stability is a more reliable signal than geometric magnitude alone. Our findings characterize Adam’s practical dominance as enabled by its approximation errors rather than despite them, trading quantifiable geometric infidelity for the computational tractability and numerical stability that make modern deep learning possible.