Methodological pitfalls in food security analytics: Addressing biases in regression, SHAP, and feature importance analyses

This paper critically examines methodological challenges in food security research through analysis of 576 articles published in the Global Food Security journal, introducing a novel "leave-top1-out" validation approach to assess feature importance reliability. Using a public dataset of 12,564 instances with 195 features, we demonstrate how supervised models achieve high prediction accuracy while producing unstable feature importance rankings. Our empirical findings reveal that unsupervised methods and non-target-prediction approaches maintain remarkable consistency in feature rankings despite perturbation, while supervised models and their SHapley Additive exPlanations (SHAP) variants exhibit dramatic shifts in feature importance hierarchies when top predictors are removed. These results substantiate our identification of primary methodological misapplications in ground truth challenges in model interpretation. The supervised models' focus on demographic characteristics rather than direct food security indicators further demonstrates how optimization for prediction accuracy can compromise reliable feature identification without consistency and dose-response relationships validation. To address these limitations, we advocate for a multifaceted analytical framework combining unsupervised techniques with nonlinear nonparametric methods to achieve greater stability in identifying reliable food security determinants. Our complete methodology and implementation code are publicly available on GitHub to promote reproducibility and methodological transparency in food security research. • Linear methods applied to nonlinear food security data produce misleading interpretations. • High prediction accuracy doesn't guarantee reliable feature importance rankings. • SHAP’s reliance on specific models can amplify biases in feature importance rankings. • Unsupervised model offers more stable feature rankings than existing ML approaches.