Gradient-based smoothing parameter estimation for neural P-splines

Abstract Due to the popularity of deep learning models there have recently been many attempts to translate generalized additive models to neural nets. Generalized additive models are usually regularized by a penalty in the loss function and the magnitude of penalization is controlled by one or more smoothing parameters. In the statistical literature these smoothing parameters are estimated by criteria such as generalized cross-validation or restricted maximum likelihood. While the estimation of the primary regression coefficients is well calibrated and investigated for neural net based additive models, the estimation of smoothing parameters is often either based on testing data (and grid search), implicitly estimated or completely neglected. In this paper, we address the issue of explicit smoothing parameter estimation in neural net-based additive models fitted via gradient-based methods, such as the well-known Adam algorithm. We therefore investigate the data-driven smoothing parameter selection via gradient-based optimization of generalized cross-validation and restricted maximum likelihood. Thus we do not need to calculate Hessian information of the smoothing parameters. As an additive model structure, we use a translation of P-splines to neural nets, so-called neural P-splines. The fitting process of neural P-splines as well as the gradient-based smoothing parameter selection are investigated in a simulation study and an application.