Ensemble projection pursuit for general nonparametric regression

Projection pursuit regression (PPR) and artificial neural networks (ANNs), both introduced around the same time, share a similar approach to approximating complex structures in data. However, PPR has received far less attention than ANNs and other popular statistical learning methods such as random forests (RF) and support vector machines (SVM). In this paper, we revisit the estimation of PPR and propose an optimal greedy algorithm and an ensemble approach via feature bagging, hereafter referred to as ePPR, to improve its effectiveness. Compared to RF, ePPR has two main advantages. First, its theoretical consistency can be proved for more general regression functions, as long as they are L2 integrable, and higher consistency rates can be achieved. Second, ePPR does not split the samples, so each term of the PPR is estimated using the entire data, making minimisation more efficient and ensuring the smoothness of the estimator. Extensive comparisons on real data sets show that ePPR is more efficient in regression and classification than RF and other competitors.