A Gradient-Norm-Aware Optimizer for Symmetry-Preserving and Stable Deep Learning

The stability of deep neural networks during training is related to the structural symmetry between gradient dynamics and parameter updates. Existing Adam-type optimizers are prone to breaking this dynamic symmetry when the gradient scale changes drastically, leading to training instability. In addition, quantitative metrics of stability are serious lacking. A novel Optimizer, Gradient-Norm-Aware Adam (GNAdam), is proposed in this paper. It introduces a gradient norm-based smooth learning rate mechanism to adjust the step size, effectively maintaining the structural symmetry and exhibiting strong training stability. A new stability evaluation metric, Stability Score Plus (SS+), combining gradient norm and parameter update norm, is designed in this paper to quantify algorithm stability. Experimental results on the CIFAR-10 dataset demonstrate that GNAdam reduces gradient ripple and parameter update instability. The stability quantification results are highly consistent with the visualization results, with the SS+ value being only 24% of Adam’s and 8% of RAdam’s. Experimental results on glaucoma datasets demonstrate that the proposed method shows significant advantages in classification metrics such as accuracy and AUC, as well as medical metrics such as sensitivity and specificity. This indicates that improved training stability can effectively translate into more reliable clinical diagnosis in complex medical image classification tasks.