A Statistics-based Feature Generation (SFG) Method: Theory and Applications

The discriminant ability of features plays a central role in classification tasks. Deep learning (DL) utilizes back-propagation to yield discriminant features with an end-to-end depth structure, which lacks interpretability. A new learning paradigm, Green Learning (GL), has been proposed to address the weaknesses of DL, including high carbon footprints, large model sizes, and mathematical opaqueness. Following this idea, a statistics-based feature generation (SFG) method is proposed to boost system performance further. SFG consists of two modules: 1) feature subset selection and 2) discriminant feature generation via linear combination. In the first module, SFG traces the splitting features along individual tree paths within auxiliary XGBoost classifiers, which form the desired feature subsets for each binary label subspace. In the second module, SFG determines the weights of features in each subspace by formulating a least-square normal equation supervised by its corresponding binary sub-labels. SFG enables the adaptive assessment of feature importance at global and various local scales. Besides, it reduces the computational requirement by working in low-dimensional feature sub-spaces. Extensive experiments are conducted on classical image classification tasks to demonstrate the effectiveness and efficiency of SFG.