Deep Learning in Reproducing Kernel Banach Spaces

Deep learning has achieved immense success in the past decade. The goal of this dissertation is to understand deep learning through the framework of reproducing kernel Banach spaces (RKBSs), which were originally proposed for promoting sparse solutions. We begin by considering learning problems in a general functional setting, and establishing explicit and data-dependent representer theorems for both minimal norm interpolation (MNI) problems and regularization problems. These theorems provide a crucial foundation for the subsequent results derived for both sparse learning and deep learning. Next, we investigate the essential properties of RKBSs capable of encouraging sparsity in learning solutions. With the proposed sufficient conditions on RKBSs, we develop sparse representer theorems by leveraging previously established representer theorems. Illustrative examples include the sequence space ℓ1(N) and a function space constructed by measures, both of which can enjoy sparse representer theorems. Subsequently, we introduce a hypothesis space for deep learning that employs deep neural networks (DNNs). By treating the DNN as a function of two variables, the physical variable and the parameter variable, we construct a vector-valued RKBS with reproducing kernel formed by the product of the DNN with a weight function. We explore MNI problems and regularization problems considering in this vector-valued RKBS, and develop the corresponding representer theorems for deep learning, demonstrating solutions in the form of DNNs. Finally, motivated by the desire of alleviating potential overfitting and accelerating inference, we examine sparse deep learning models with ℓ1 regularization, which encourages a significant portion of parameters in DNNs to be zero. We provide an iterative algorithm for selecting regularization parameters to ensure that the weights in each layer of the resulting sparse DNN meet a specified sparsity level. Numerical experiments validate the efficiency of the proposed algorithm in acquiring appropriate regularization parameters to achieve desirable sparse DNNs.