How Missingness Mechanism and Imputation Strategy Jointly Shape the Stability of Post-Hoc Feature Attributions: A Cross-Model Empirical Study

Post-hoc feature-attribution methods such as SHAP and permutation importance are widely used to explain the predictions of machine learning models trained on tabular data, yet real-world tabular datasets are rarely complete: values go missing according to different underlying mechanisms, and practitioners routinely impute them before scoring or explaining a model. Prior work has shown that the choice of imputation method can bias SHAP values, but this line of work has generally examined a single missingness mechanism and a single attribution method in isolation. It remains unclear whether the mechanism generating the missingness (Missing Completely at Random, MCAR; Missing at Random, MAR; or Missing Not at Random, MNAR) interacts with the choice of imputation strategy, whether such effects are consistent across model families, and whether different attribution methods (SHAP versus permutation importance) reach the same practical conclusions about which imputation strategy is safest. We address these questions with a controlled empirical study on two public tabular benchmarks (the Breast Cancer Wisconsin diagnostic dataset and the Pima Indians Diabetes dataset), amputing held-out test data under all three missingness mechanisms at three missing-data rates (10%, 20%, 30%), imputing with three established strategies (mean/median, k-nearest-neighbours, and iterative/MICE-style imputation), and re-scoring three frozen model families (logistic regression, random forest, and LightGBM) trained on the complete data. We measure attribution stability as the Spearman rank correlation between the SHAP or permutation-importance ranking obtained under each imputed scenario and the ranking obtained on the complete-data oracle test set, using 978 experimental conditions with repeated random masking and non-parametric statistical testing (Friedman, Wilcoxon signed-rank with Holm correction, Kendall's tau, and bootstrap confidence intervals). We find that (i) missingness mechanism has a statistically significant but numerically small effect on SHAP stability, with MNAR producing significantly lower stability than MCAR (Holmcorrected p = 0.0042); (ii) imputation method has a large and highly significant effect (Friedman chisquare = 52.4, p = 4.1e-12), with mean/median imputation producing markedly less stable SHAP rankings than KNN or iterative imputation, but this effect is concentrated almost entirely in LightGBM and is negligible for logistic regression; (iii) SHAP-based rankings are dramatically more stable under imputation than permutation-importance rankings (median Spearman correlation approximately 1.00 versus 0.90, with permutation importance showing a long tail of near-zero correlations); and (iv) SHAPbased and permutation-based attribution methods agree on which imputation strategy is best for explanation stability in only 33% of dataset-model-mechanism combinations (mean Kendall's tau = 0.15), meaning that the practical recommendation a practitioner would derive depends on which attribution method they use to evaluate it. These results suggest that explanation-stability audits under missing data should be attribution-method-specific and model-family-specific rather than treated as a single generic property of a dataset, and that permutation importance in particular should be interpreted cautiously whenever the scored features have been imputed.