An ESMValTool-based framework for sanity checks, physical consistency and climate fidelity during model development – ICONEval v1.0

Continuous evaluation and performance monitoring during the development of Earth System Models (ESMs) are essential to identify potential problems early, such as unrealistic behavior of climate-relevant quantities, insufficient skill in reproducing the observed basic climate state, or violations of physical laws. The latter is particularly important for the emerging class of hybrid machine learning (ML) enhanced ESMs, where data-driven components are integrated with physics-based model formulations. ESMs used for projections of future climate continue to increase in complexity and resolution. Efficient and user-friendly tools such as the Earth System Model Evaluation Tool (ESMValTool) can therefore greatly support the assessment of a model. So far, ESMValTool focused primarily on providing a broad collection of community-developed evaluation diagnostics and recipes, allowing users to perform a large variety of rather detailed assessments across different domains. A main application of the tool was the assessment of multiple ESMs, in particular those participating in the coupled model intercomparison project (CMIP). Here, we introduce ICONEval, an open-source evaluation framework using ESMValTool that complements existing capabilities by enabling rapid, reproducible, and physically informed assessments of model performance, also during development. ICONEval provides efficient parallel processing of ESMValTool recipes and can generate HTML summary reports allowing to easily automatize and visualize evaluation and monitoring of performance during model development. The new capabilities are grouped into three complementary categories: (1) sanity checks, (2) physical consistency checks, and (3) climate fidelity diagnostics. The sanity checks assess whether global mean values of climate-relevant variables are within the bounds derived from observational and reanalysis datasets. The physical consistency checks aim to identify potential violations of constraints imposed by fundamental physics such as conservation of total air mass, realistic variability of atmospheric water vapor with temperature or the temperature dependence of the cloud ice fraction. The climate fidelity diagnostics assess important climate variables from different ESM components, i.e. atmosphere, ocean, and land. Here, we demonstrate this extension of the ESMValTool capabilities by applying the new diagnostics to a historical simulation performed with the ICON-XPP model as an illustrative example. The three-step assessment presented here can be efficiently used to compare different model configurations or versions, for example when testing new or updated parameterizations, including hybrid ML-enhanced (MLe) ESMs, also supporting emerging community benchmarking standards such as ClimateBench. While ICONEval is focused on the ICON model, the underlying ESMValTool diagnostics can also be used for different models as long as the model output is following the CMOR (Climate Model Output Rewriter) standard or the model is supported directly by ESMValTool.