AI Model Performances Depend on Adequate Training Data Representing Diverse Physical Mechanisms

Abstract The value of weather and climate AI models depends on their generalization beyond training data, yet the influence of data characteristics on this capability remains unclear. In this investigation, a fully controllable framework using a parameter‐tunable Burgers system is utilized to explore the role of data in model generalization. By varying nonlinear and diffusive coefficients, 600 distinct dynamical regimes are generated, and two experiments are performed using samples drawn from different parameter quadrant or their combinations. The results demonstrate generalization depends on adequate training data representation of underlying mechanisms of target regimes, including both effective Reynolds numbers and solution properties. Insufficient representation, either through missing mechanisms or low sample proportions of certain mechanisms, can lead to substantial performance degradation. These findings imply AI models trained on historical reanalysis may perform poorly in extremes and climate‐scale predictions, requiring a pressing need to supplement samples with physically consistent numerical simulations.