Phylogenetically Structured Validation of Machine Learning Models for Primate Threat Status Assessment

Machine learning models have been employed increasingly as tools to inform conservation assessment when ecological data is insufficient; however, conventional random validation can lead to an overestimate of predictive transferability when closely related species are included in training and testing datasets. In this paper, the effect of the validation framework on predicting the current threat status of primates using a machine-learning approach is investigated. The data set consists of 423 primate species whose status was determined by the IUCN Red List and were categorized as either being threatened or non-threatened along with their ecological, habitat, behavioral, biogeographic, and phylogenetic attributes. A fixed 100-tree Random Forest model is evaluated using random stratified, genus-blocked, family-blocked, and phylogenetically blocked validation frameworks using 100 posterior mammal trees. Averaged ROC-AUC values are 0.849 for random validation, 0.829 for unseen genera, 0.768 for unseen families, and approximately 0.73 for the main phylogenetic blocking. Sensitivity analyses on different phylogenies, levels of block coarseness, approaches to missing data imputation, classification criteria, and forest stability, as well as 360 simulated data sets with increased validation optimization, have been conducted.