Analysis of Neural Network Regularization Mechanism Based on Lagrangian Optimization

Overfitting problems exist in neural networks, which have greatly restricted the performance of learning. In this paper, a regularization mechanism based on Lagrangian optimization is proposed. A Lagrangian function is constructed by integrating two regularization terms: sparsity regularization and smooth regularization. Sparsity regularization is a method that makes some elements of neural network parameter matrix into zero, so as to simplify the model structure. In this paper, smooth regularization focuses on the relationship between parameters so as to improve the stability and generalization ability. The model complexity can be controlled dynamically by incorporating this function into the neural network loss function. MNIST, CIFAR-10 and ImageNet have proved its superiority. On MNIST, this model achieves 99.2 % test precision, 1.5 % higher than L1 regularization, 1.2 % higher than <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$L 2$</tex> regularization, and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$0.04,0.08$</tex> less than <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$L 2$</tex> regularization. On CIFAR-10, the test precision is 88.6 %, 3.2% and 2.8 % higher than that of L1 and L2 regularization, and the test loss is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{0. 1 5}$</tex>. On ImageNet, a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{2. 1 \%}$</tex> increase in Top-1 accuracy and a 12 % reduction in training time. Compared to classical regularization methods, this method can balance both sparsity and smoothness constraints in a dynamic way, which can significantly reduce overfitting while improving generalization efficiency.