Score Matching with Deep Neural Networks: A Non-Asymptotic Analysis

Score matching is a statistical approach for estimating the score (the gradient of the log-density) of a probability distribution from samples. It has found a number of applications, including in generative modeling, where it serves as a key component of the state-of-the-art diffusion modeling framework. The goal of this work is to provide non-asymptotic bounds on the score matching risk in the setting where the score model is a deep neural network. Here key challenges include the fact that the score model is vector-valued and that the score matching loss depends on the Jacobian of the score model. Our approach integrates results from empirical process theory, including classical bounds and recently introduced techniques for bounding covering numbers of neural network models, with novel covering results to address these challenges. The resulting bound has logarithmic dependence on the network width, allowing the network size to grow exponentially with the number of training samples without compromising the bound.