Predicting the dataset-level utility of feature-selection portfolio search

Feature selection (FS) can improve prediction, while searching many combinations of selectors, subset sizes, and classifiers adds substantial cost. We predict the dataset-level AUC gain from the best combination in this FS-enabled portfolio over the best all-feature classifier. Among 99 datasets in the prediction sample, 51 show any positive gain and five gain at least 0.04 absolute AUC points. A class-weighted Random Forest using five basic descriptors and 21 Problexity measures achieves leave-one-dataset-out AUROC of 0.722 for any positive gain and 0.708 for gains of at least 0.04. Matched random-subset searches and outer-fold tests provide complementary evidence on search utility. SHAP analysis identifies neighborhood structure, feature overlap, and dataset size as influential predictors. These results support dataset-level prioritization of FS portfolio search and identify directions for strengthening its predictive precision.