Validation-Aware AI for Soil Contamination Assessment and Remediation

Machine learning models for mapping heavy metal concentrations in soil are becoming increasingly popular for making decisions regarding which areas to remediate. However, the reported accuracies of these models are typically based upon the same set of clustered samples that were used to create the model. In this study, we use a controlled simulation to determine how reliable the reported accuracies are and what the implications are for remediation decisions. Twenty synthetic lead contaminated sites (each 100 x 100 grid) were created with known concentrations in each cell. Each site was then sampled using preferential clustered sampling methods to obtain 300 laboratory samples. Eight different model configurations were evaluated using both random and spatial-block cross-validation against a complete census of all un-sampled cells. Random cross-validation overestimated the census R2 for 19 out of 20 sites (mean 0.86 vs. 0.74), while spatial-block cross-validation was pessimistic (0.66). Prediction intervals calibrated using random residuals provided 82% coverage at a nominal 90%, and 64% outside of the area of applicability; spatial-block calibration provided 94% and 84%. By treating each prediction as a probability and applying a cost-ratio rule, we were able to reduce missed exceedances from 31% to 5% at a miss-to-false-alarm cost ratio of 10. Additional space-filling sampling resulted in the lowest map error, while preferential and uncertainty-driven sampling resulted in the highest recall of contaminated cells. Models developed solely from spectral proxies did not transfer well to new sites. These results suggest that reporting accuracy under multiple validation schemes, utilizing uncertainty-aware decision rules, and clearly defining the limits of applicability will improve decision-making. All findings are simulation-based and will need to be verified with field data. Keywords: Spatial cross-validation; Conformal prediction; Area of applicability; Heavy-metal mapping; Risk-based decision-making