Mean activation curvature for scalable second-order optimization in deep networks

Abstract Second-order methods can accelerate deep neural network training, but their adoption is limited by the cost and instability of estimating and inverting curvature matrices. We revisit Kronecker-factored Fisher approximations via an empirical structural analysis of activation and gradient statistics in modern architectures. Across a range of models, we find that activation statistics capture most of the effective curvature directions, while gradient statistics mainly act as a global or diagonal rescaling. Based on this observation, we propose mean activation curvature , a scalable curvature surrogate that yields two optimizers, MAC and SMAC , offering different trade-offs between expressiveness and efficiency. We further extend the construction to self-attention layers with a structured approximation that retains the role of attention scores, and provide a convergence analysis under standard assumptions. Experiments on vision and language models show that MAC / SMAC matches or improves the accuracy of existing second-order baselines while reducing training time and memory usage. Code: github.com/hseung88/mac .