Evaluation of Weather Forecasts from AI Big Models over East Asia

Reliable medium-range weather forecasts are crucial for both science and society. Although weather predictions primarily rely on numerical weather models, the artificial intelligence (AI) weather big models have shown potential for accurate weather forecasts with reduced computational costs. However, prediction skills of big models remain uncertain, particularly in East Asia, which limits further application of weather AI models. To systematically evaluate predictive capabilities of Pangu, FuXi, and GraphCast models over East Asia, their prediction results are focusing on 500 hPa geopotential height, 2 m air temperature, 10 m wind speed, precipitation, and track of tropical cyclones.ECMWF reanalysis V5 (ERA5) datasets are utilized to provide the initial conditions for big models, and to assess their predictive skill. Additionally, precipitation observations and China Meteorological Administration tropical cyclone datasets are utilized to access big models as well. FuXi shows the highest forecasting skills among 3 big models for 500 hPa geopotential height. The forecast from FuXi is reliable for up to 9.75 days, while the forecasts from Pangu and GraphCast are reliable for 8.75 days and 8.5 days, respectively. For 2 m air temperature forecasting, FuXi presents higher skills with an averaged temporal anomaly correlation coefficient (TCC) ranging from 0.48 to 0.91, while TCCs of Pangu and GraphCast are 0.43-0.91 and 0.38-0.83, respectively. Among 3 models, only FuXi and GraphCast provide precipitation forecasts. FuXi shows higher prediction skill compared to GraphCast in forecasting precipitation, light rain, and moderate rain; however, GraphCast has advantage in heavy rain forecast. As the lead time increases, the threat scores (TSs) of FuXi for rainfall, light rainfall and moderate rainfall are 0.22-0.41, 0.15-0.24 and 0.06-0.22, respectively. The model demonstrates higher skill in the northern and southeastern regions of China. For predicting the track of cyclones, Pangu model demonstrates superior predictive skill. As the lead time increases from 6 hours to 240 hours, biases of Pangu's prediction track increase from 17.5 km to 1850 km.The study focuses on the prediction skill of various AI big models through TCC, spatial anomaly correlation coefficient, and TS. Generally, the performance of FuXi is superior for most elements. And reasonable evaluation of AI model is helpful for the development of AI models.