VSI: An interpretable Bayesian feature selection method based on Vendi score

Practitioners use various feature importance metrics to rank features based on their importance to discard weak predictors, simplifying predictive models and improving generalizability. Model-dependent feature ranking methods often rely on training a machine learning model. However, a common issue is the confusion between feature importance and feature impact on the trained model, leading to potential misinterpretations in fields such as medicine and business. This problem primarily stems from the widespread use of model-based methods that leads to misinterpreting feature importance with its impact on the trained model. To address this, we introduce Vendi Score Importance (VSI), an interpretable model-independent feature ranking metric. VSI quantifies each feature’s impact on the overall Vendi score instead of a machine learning model, beginning with only the target/label and sequentially adding features. As a model-independent metric, VSI relies solely on feature and target values, making it applicable to both classification and regression tasks. We demonstrate VSI’s effectiveness through applications to four benchmark problems (two classification and two regression), comparing it to LIME, Permutation, and SHAP (model-agnostic methods), as well as Mutual Information Regression (MIR), a univariate, model-independent approach. Results show that VSI not only provides interpretable importance scores for each feature but also outperforms others in computational efficiency. To confirm VSI’s robustness against random noise, we repeated the feature ranking procedure 100 times, introducing different random features each time. The results highlight VSI’s superior performance, particularly in maintaining agnosticism to random noise.