HEFS-EXR: A Robust Hybrid Ensemble Feature Selection Algorithm

As the volume of data continues to grow, the performance of predictive models can be adversely affected by the presence of largely irrelevant and redundant features. This is further amplified when working with small-sample data, where the risk of overfitting becomes more pronounced. In such settings, feature selection emerges as a critical component of the machine learning process, enabling improved predictive performance, stability, feature quality, and greater explainability, particularly in ``high-stakes'' domains. Existing feature selection approaches are numerous; however, ensemble learning-based feature selection methods remain comparatively scarce, despite their potential to improve predictive performance, stability, and generalizability. Moreover, no single method consistently balances predictive performance, stability, feature quality, and explainability across diverse problem settings. To address these limitations, we introduce the HEFS-EXR method, a two-stage hybrid ensemble feature selection algorithm that automatically determines the feature subset size at each stage. The method is designed to leverage the strengths of multiple feature selection methods. We conducted experiments on 8 synthetic and 22 real-world datasets to evaluate our approach. Our findings indicate that HEFS-EXR consistently produces compact feature subsets, achieving dimensionality reductions ranging from 14% to 93% across both synthetic and real-world data from diverse domains, without compromising predictive performance and, in many cases, improving it. The results also demonstrated improved stability, highlighting the method's robustness.