Bringing cross-validation into the real world to evaluate transferability of satellite-based vegetation models

Near-real-time mapping of vegetation using satellite imagery is becoming increasingly common and valuable across a wide range of ecosystems. The availability of large datasets has led many researchers to complex machine learning algorithms (MLAs) to train satellite models. However, complex MLAs may underperform for the inherently extrapolative applications required for real-world vegetation monitoring. We used a dataset of nearly 10,000 training samples of standing herbaceous grazingland biomass collected over ten years to train progressively more complex MLAs, test them across progressively more extrapolative cross-validation (CV) groupings, and evaluate their transferability and consistency. The performance of all MLA's decreased substantially when tested against more extrapolative CV groupings. The commonly used approach of random k-fold CV produced overly optimistic performance (R²: 0.71-0.78) compared to a more realistic task of predicting for an unseen year (R²: 0.49-0.54). Simpler MLAs, such as partial least squares regression, were more consistent and outperformed complex MLAs for the most extrapolative tasks, and performance was less sensitive to the distinctness of unseen test data. We conclude that random k-fold CV likely produces unrealistically optimistic expectations for real-world applications of satellite vegetation models, and could be associated with major prediction misses when models are used in novel environmental conditions.