AdamN: Accelerating Deep Learning Training via Nested Momentum and Exact Bias Handling

This paper introduces AdamN, a nested-momentum adaptive optimizer that replaces the single Exponential Moving Average (EMA) numerator in Adam/AdamW with a compounded EMA of gradients plus an EMA of that EMA, paired with an exact double-EMA bias correction. This yields a smoother, curvature-aware search direction at essentially first-order cost, with longer, more faithful gradient-history memory and a stable, warmup-free start. Under comparable wall-clock time per epoch, AdamN matches AdamW’s final accuracy on ResNet-18/CIFAR-100, while reaching 80% and 90% training-accuracy milestones ~127 s and ~165 s earlier, respectively. On pre-benchmarking workloads (toy problems and CIFAR-10), AdamN shows the same pattern: faster early-phase convergence with similar or slightly better final accuracy. On language modeling with token-frequency imbalance—Wikitext-2-style data with training-only token corruption and a 10% low-resource variant—AdamN lowers rare-token perplexity versus AdamW without warmup while matching head and mid-frequency performance. In full fine-tuning of Llama 3.1–8B on a small dataset, AdamN reaches AdamW’s final perplexity in roughly half the steps (≈2.25× faster time-to-quality). Finally, on a ViT-Base/16 transferred to CIFAR-100 (batch size 256), AdamN achieves 88.8% test accuracy vs. 84.2% for AdamW and reaches 40–80% validation-accuracy milestones in the first epoch (AdamW reaches 80% by epoch 59), reducing epochs, energy use, and cost.