Recommendations for Comprehensive and Independent Evaluation of Machine Learning‐Based Earth System Models

Abstract Machine learning (ML) is a revolutionary technology with demonstrable applications across multiple disciplines. Within the Earth science community, ML has been most visible for weather forecasting, producing forecasts that rival modern physics‐based models. Given the importance of deepening our understanding and improving predictions of the Earth system on all time scales, efforts are now underway to develop Earth‐system models (ESMs) capable of representing all components of the coupled Earth system (or their aggregated behavior) and their response to external changes over long timescales. Building trust in ESMs is a much more difficult problem than for weather forecast models, not least because the model must represent the alternate (e.g., future or paleoclimatic) coupled states of the system for which there are no direct observations. Given that the physical principles that enable predictions about the response of the Earth system are often not explicitly coded in these ML‐based models, demonstrating the credibility of ML‐based ESMs thus requires us to build evidence of their consistency with the physical system. To this end, this paper puts forward five recommendations to enhance comprehensive, standardized, and independent evaluation of ML‐based ESMs to strengthen their credibility and promote their wider use.

Authors

Institutions

Publication Details

Journal
Journal of Geophysical Research Machine Learning and Computation
Published
2025-03-01
DOI
https://doi.org/10.1029/2024jh000496
Citations
6
Primary Topic
Climate variability and models
Type
article
Field-Weighted Citation Impact
6.63

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Recommendations for Comprehensive and Independent Evaluation of Machine Learning‐Based Earth System Models

Paul Ullrich, William D. Collins, María J. Molina, Katherine Dagon et al.
6 citations
Journal of Geophysical Research Machine Learning and Computation
Climate variability and models
6.63
article

Recommendations for Comprehensive and Independent Evaluation of Machine Learning‐Based Earth System Models

Paul Ullrich, William D. Collins, María J. Molina, Katherine Dagon, Shiheng Duan, Elizabeth A. Barnes, Dan Lu, Joe D. Elms, Travis O’Brien, Lai-Yung Ruby Leung, Jiwoo Lee, F. O. Rebassoo
article en
6 citations

Abstract

Abstract Machine learning (ML) is a revolutionary technology with demonstrable applications across multiple disciplines. Within the Earth science community, ML has been most visible for weather forecasting, producing forecasts that rival modern physics‐based models. Given the importance of deepening our understanding and improving predictions of the Earth system on all time scales, efforts are now underway to develop Earth‐system models (ESMs) capable of representing all components of the coupled Earth system (or their aggregated behavior) and their response to external changes over long timescales. Building trust in ESMs is a much more difficult problem than for weather forecast models, not least because the model must represent the alternate (e.g., future or paleoclimatic) coupled states of the system for which there are no direct observations. Given that the physical principles that enable predictions about the response of the Earth system are often not explicitly coded in these ML‐based models, demonstrating the credibility of ML‐based ESMs thus requires us to build evidence of their consistency with the physical system. To this end, this paper puts forward five recommendations to enhance comprehensive, standardized, and independent evaluation of ML‐based ESMs to strengthen their credibility and promote their wider use.

Journal of Geophysical Research Machine Learning and ComputationVol. 2(1)
NSF National Center for Atmospheric Research (US), Lawrence Livermore National Laboratory (US), Oak Ridge National Laboratory (US), Planetary Science Institute (US), Pacific Northwest National Laboratory (US), Lawrence Berkeley National Laboratory (US), Indiana University Bloomington (US), NSF NCAR Climate and Global Dynamics Laboratory (US), University of Maryland, College Park (US), University of California, Davis (US), Colorado State University (US), University of California, Berkeley (US)
National Science Foundation, U.S. Department of Energy, Battelle, Eli Lilly and Company, National Center for Atmospheric Research, Indiana University, Lilly Endowment, Global Down Syndrome Foundation, Office of Science, Environmental Resilience Institute, Indiana University, Biological and Environmental Research, Laboratory Directed Research and Development, Lawrence Livermore National Laboratory, Oak Ridge National Laboratory, Lawrence Berkeley National Laboratory, Pacific Northwest National Laboratory
Openalex Percentile: Top 3%
Climate variability and models
6.63
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.