Multimodal Ultra-Short-Term Irradiance Forecasting Based on Sky Images: Exploring the Complementary Characteristics of Clouds and Irradiance

The high uncertainty in solar irradiance changes limits its application in power grids. Data-driven research that integrates sky images and irradiance has made significant advances. However, there has been limited attention to the impact of different fusion methods for sky images on prediction performance. We propose a novel multi-modal interaction fusion framework aimed at extracting complementary correlations between these two heterogeneous data sources to improve the accuracy of predicting critical ramp events. First, 3D-ResNet and Informer are used to encode the spatial features of sky images and the temporal features of irradiance, respectively. An interaction matrix is then constructed to fuse these two modalities, ultimately generating solar irradiance predictions. Experimental results show that, compared to state-of-the-art baseline models, the proposed method reduces RMSE by 20.9 %, enhancing the model's ability to leverage information from sky images.