Low‐Dimensional Adaptive Neural Network Regression With Directional Change Detection via Nuclear Norm Penalization

ABSTRACT This paper reports on our study of a regularized neural network regression method that adapts to the low‐dimensional directional structure of a function. The key innovation lies in identifying direction vectors where the function exhibits significant variation and conducting estimation within this reduced‐dimensional space. This is achieved through a regularization scheme that controls the nuclear norm of the weight matrix. The nuclear norm penalization approach effectively reduces its rank and allows the model to discover the principal subspace where the function varies. Additionally, an penalty is imposed to introduce node‐level sparsity, which further enhances dimension reduction and improves estimation efficiency. The combination of the nuclear norm and norm penalties results in a low‐dimensional network structure that strikes a good balance in the bias‐variance trade‐off and improves estimator performance. An efficient and stable implementation scheme is designed based on the alternating direction method of multipliers and the Levenberg–Marquardt algorithm. The stability of the algorithm is enhanced by using a B‐spline activation function with compact support and an initialization strategy based on model‐based sliced inverse regression. Numerical experiments on simulated and benchmark datasets demonstrate that our method outperforms several popular machine learning and neural network regression techniques.