Nonparametric optimality for large compressible deep neural networks under quadratic loss functions

Establishing theoretical analysis that explains the empirical success of deep learning has attracted increasing attention in modern machine learning literature. A large amount of existing research focuses on the advantages of deep neural networks in a specific context, which does not explain the wide range of neural network applications. In a class of physically verifiable situations, this paper aims to evaluate the excess risk of a deep learning estimator based on feed-forward neural networks with the ReLU activation function. In particular, we establish a unified theoretical analysis for the optimal excess bounds under the quadratic loss and the composite structures of the true function. The obtained excess bounds are built upon so-called compressible 条件 on over-parameterized neural networks, including widely-used sparse networks, and low-rank networks associated with weight matrices as special cases. To the best of our knowledge, the current work is the first to introduce compressible 条件 to study the excess risk of neural network estimation and the first to theoretically analyze the estimation error of low-rank neural networks.