System Summarization Based on Multimodal Language Model with Attention-Weighted Fusion

Power system briefings allow grid dispatchers to quickly understand the operating status of the power system and provide decision support, therefore the generation of power system briefings is of great importance to the stable operation of the power system. Although existing summary generation studies have made some progress, most of them have not focused on the power system domain, especially the utilization of visual information is insufficient. In order to fully utilize the multimodal information in power system, we propose a new multimodal summary generation model based on attention-weighted fusion in conjunction with BART pre-trained language model. In order to overcome the problem of scarcity of multimodal datasets in the power system domain, the power system briefing generation model is trained using a transfer learning approach, and the structure of forgetting gates is introduced to better capture the visual features. Extensive experiments and comparisons with state-of-the-art methods on multimodal datasets of power systems have validated the effectiveness of the proposed approach.