Invariance‐Embedded Machine Learning Models for Sub‐Grid Scale Stress in Meso‐Scale Hurricane Boundary Layer Flows: Basic Model and A priori Assessment

ABSTRACT This study develops invariance‐embedded machine learning (ML) sub‐grid‐scale (SGS) stress models admitting turbulence kinetic energy (TKE) backscatter toward more accurate large eddy simulation (LES) of meso‐scale turbulent hurricane boundary layer flows. The new ML SGS model consists of two parts: a classification model used to distinguish regions with either strong energy cascade or energy backscatter from those with mild TKE transfer and a regression model used to calculate SGS stresses in regions with strong TKE transfer. To facilitate model implementation in computational fluid dynamics (CFD) solvers, the Smagorinsky model with a signed coefficient , where a positive value indicates energy cascade and a negative one indicates energy backscatter, is employed as the carrier for the ML model. To improve its robustness and generality, both physical and geometric invariance features of turbulent flows are embedded in the model input for classification and regression, and the signed Smagorinsky model coefficient is used as the regression model's output. All model assessments in this work are carried out in an a priori setting against filtered high‐fidelity data. Specifically, various machine‐learning methods and input configurations have been used to evaluate the classification model's performance. The F1‐scores, which measure balanced precision and recall, of classification models with embedded physical and geometric invariance can be improved by about over those without geometric invariance. Regression models based on ensemble neural networks have demonstrated superior performance in predicting the signed Smagorinsky model coefficient, exceeding that of the dynamic Smagorinsky model (DSM) in a priori tests.