Tropical Cyclone Image Super-Resolution via Multimodality Fusion

The traditional super-resolution dataset construction using artificial down-sampling techniques can result in information loss, insufficient diversity, and non-uniqueness. Furthermore, existing methods for image super-resolution are limited to single-modal images and cannot accommodate the complexities of multimodal images. This is problematic because diverse modal data requires individualized model design and training, which can hinder the exploitation of complementary relationships among multimodal data. In this article, we have addressed these issues by undertaking a two-step solution approach. In the first step, we constructed a super-resolution dataset that utilized remote-sensing images of tropical cyclones in “real cases.” This dataset comprises HR–LR image pairs originating from multiple sensors of varying satellite sources, resulting in multimodal data. However, the HR–LR image pairs suffer from an additional misalignment issue. Thus, in the second step, we designed a super-resolution network based on MAT to address the misalignment problem in multimodal environment. After numerous ablation experiments and comparison experiments, we have shown that our model is effective, with an improvement of 50% over the original baseline model, and an increase varying between 20% and 50% compared to other common super-resolution models. We have made our source code and data publicly available online at https://github.com/kleenY/MMTCSR .