Multimodal data fusion by temporal tensor networks for tropical cyclone intensity prediction

The prediction of the intensity of tropical cyclones (TCs) is crucial for weather forecasts and disaster prevention. Maximum sustained wind (MSW) is one of the main indexes of TC intensity. Analyzing multispectral images (MSIs) of cyclones by deep learning methods can increase MSW accuracy. However, existing methods are mostly designed for infrared images, not being able to leverage different band data or represent the rich temporal-spectral-spatial features in MSIs. Meanwhile, MSIs alone cannot provide all the necessary or accurate information of TCs as there are usually undesired variations or distortions in TC structures reflected by the images due to the uniqueness of each TC and positions of TCs and the satellites that capture images. Moreover, TC formation and evolution are affected by various physical factors that are not recorded in MSIs or cannot be easily derived from the images. To perform multimodal data fusion while making use of the most valuable information, we propose a novel model, Invalid-Band-Suppressed and Structure-Descriptor-Enhanced Temporal Tensor Network (ISSDTN). ISSDTN extracts features from the long-wavelength and the short-wavelength band images of each set of TC MSIs in two separate paths to suppress invalid band data. Finally, the paths are combined to fuse the multimodal information, i.e., image features and Structure Descriptors (SDs) via cross-attentions to predict MSW. Experimental results show that ISSDTN outperforms many baselines and state-of-the-art methods in various cyclone datasets. The errors of 24 h MSW prediction by ISSDTN is as low as 4.49 m/s and 5.33 m/s for FY4A-TC and TCIR datasets, respectively.