Towards Hybrid Physics-Machine Learning Parameterizations: Employing Data Assimilation for Online Learning of Turbulence and Convection Closures in a Unified Scheme

Despite advances in climate modeling, the spread in equilibrium climate sensitivity estimates has remained largely unchanged over generations of modeling, mainly due to uncertainties in cloud feedback mechanisms arising from subgrid-scale turbulence, convection, clouds, and the resulting cloud-radiation interactions. Misrepresentations of these processes affect both long-term climate projections and the simulation of short-term atmospheric phenomena, such as the diurnal cycle of precipitation. These limitations are most pronounced in regimes like stratocumulus clouds and their transition to cumulus over ocean basins---areas where climate models have the largest cloud biases in the historical record. This thesis aims to constrain the critical subgrid-scale physics of turbulence and convection by developing and calibrating a hybrid physics–machine learning parameterization using the Eddy-diffusivity Mass-flux (EDMF) framework. By integrating machine learning components into the EDMF and employing data assimilation techniques for online learning, we attempt to directly target some of the processes responsible for uncertainties in cloud feedbacks. In this thesis, we employ ensemble Kalman inversion within a single-column setup to simultaneously perform online calibration of parameters in empirical closures and embedded neural networks, targeting large-eddy simulations as ground truth. The online learning framework ensures stability and physical consistency, as machine learning components are trained within the context of the full model dynamics. By directly targeting poorly constrained processes like lateral entrainment/detrainment and turbulent mixing lengths, we improve the representation of subgrid-scale fluxes and resulting cloud properties across various atmospheric regimes. We uncover limitations of traditional semi-empirical closures, providing insights for future model development. The calibrated hybrid parameterization outperforms existing schemes, particularly in regions where climate models have historically underperformed, and maintains accuracy in out-of-sample forcings from a warmer climate. This work demonstrates that integrating machine learning with physics-based parameterizations through data assimilation offers a systematic and robust approach for reducing biases in climate models and understanding the physics of elusive subgrid-scale closures.