Analysis and Local Convergence Proof of a Constrained Optimization Algorithm for Training Neural Networks

Training neural networks with constraints by solving a constrained optimization problem has the potential to extend their applicability. Promising results have been obtained using approaches based on the augmented Lagrangian and the primal-dual method. This work offers a concise self-contained presentation of the relevant optimization theory. The main result is a proof of local convergence of an algorithm that integrates the primal-dual method with Adam optimizer. Numerical experiments are used to validate the theory and provide practical insights.