Analysis of Multi Grade Deep Learning

Multi-Grade Deep Learning (MGDL) is a training framework that incrementally builds deep neural networks. It does this by dividing the training process into multiple “grades,” where each grade sequentially trains a shallow neural network to learn the residue from the previous one, using the outputs of prior grades as input. This approach progresses from shallow to deep architectures. This dissertation offers a comprehensive theoretical and numerical analysis of the MGDL methodology. We first demonstrate that MGDL can effectively learn target functions within the sum-composition learning format. In this context, MGDL approximates high-frequency components by composing multiple low-frequency functions. This unique capability explains how MGDL addresses the spectral bias commonly observed in standard deep neural networks, which typically struggle to learn high-frequency features. Extensive numerical experiments confirm that MGDL significantly improves the learning of these high-frequency components in target functions. Next, we analyze the gradient descent method for training deep neural networks and highlight MGDL’s advantages over standard architectures. Specifically, we show that when each “grade” in MGDL consists of a single hidden layer with a piecewise linear activation function, it transforms a highly nonconvex optimization problem into a sequence of convex subproblems. This transformation provides a theoretical explanation for MGDL’s improved trainability. We then apply MGDL to image reconstruction tasks, proposing a proximal gradient method to solve the resulting optimization problem and providing a convergence analysis of the algorithm. Experimental results indicate that MGDL achieves more stable training compared to standard deep neural networks. To understand this enhanced stability, we conduct an eigenvalue analysis of the iterative matrices arising from gradient descent. This analysis reveals that training stability is governed by the distribution of these eigenvalues. Numerical experiments show that standard deep neural networks often produce eigenvalues outside the convergence region, while MGDL consistently maintains eigenvalues within it, thereby explaining its superior stability