Successive affine learning for deep neural networks

This paper introduces the Successive Affine Learning (SAL) model for constructing deep neural networks (DNNs). Traditionally, a DNN is trained by solving a single non-convex optimization problem, which is often challenging due to its high non-convexity and large number of layers. To address this challenge, the Multi-Grade Deep Learning (MGDL) model was recently initiated by the author of this paper, inspired by the human education system. MGDL constructs a DNN progressively, training a sequence of shallow networks with fewer layers. However, it still requires solving multiple non-convex optimization problems. The proposed SAL model evolves from MGDL by leveraging the structure of DNNs, where each layer consists of an affine transformation followed by an activation function. In SAL, we train the affine transformation first by solving a quadratic/convex optimization problem, incorporating the activation function only after learning the weight matrix and bias vector for the current layer. In the context of function approximation, given a target function, the SAL model generates an expansion of the function with adaptive basis functions in the form of DNNs. We establish key theoretical results, including the Pythagorean identity and the Parseval identity for the system generated by SAL. Furthermore, we prove a convergence theorem, showing that the SAL process either terminates after a finite number of grades or ensures a strictly decreasing sequence of optimal error norms, converging to a limit as the number of grades approaches infinity. Finally, we present numerical experiments as a proof of concept, demonstrating that the SAL model significantly outperforms traditional deep learning approaches