Convergence analysis and application for high-order neural networks based on gradient descent learning algorithm via smooth regularization

The pi-sigma network (PSN), as a high-order network, has demonstrated its capacity for rapid learning and strong nonlinear processing. This paper proposes a type algorithm, for PSNs using a batch gradient method based on [Formula: see text] regularization. Direct application of [Formula: see text] regularization during network training presents two main drawbacks. There are numerical oscillations and theoretical challenges in computing the gradients at the origin. We then introduced smoothing functions by approximating the [Formula: see text] regularization to overcome these obstacles, resulting in a new gradient descent method based on smoothing [Formula: see text] regularization (GDS[Formula: see text]). Numerical results for the 4-dimensional parity problem and the nonlinear Gabor function problem demonstrate that the GDS[Formula: see text] algorithms perform better than the other four regularization methods in terms of generalization and pruning efficiency. Theoretical analysis and experimental verification strictly prove the monotonicity and strong and weak convergence of the network based on the GDS[Formula: see text] algorithms.