CoReFuNet: A coarse-to-fine registration and fusion network for typhoon intensity classification using multimodal satellite imagery

Typhoons cause significant damage to coastal and inland areas, making the accurate classification of typhoon cloud images essential for effective monitoring and forecasting. While integrating multimodal data from different satellites can improve classification accuracy, existing methods often rely on aligned images and fail to account for radiometric and structural differences, leading to performance degradation during image fusion. In registration methods designed to address this issue, two-stage approaches inaccurately estimate deformation fields, while one-stage methods typically overlook radiometric differences between typhoon cloud images. Additionally, fusion methods suffer from inherent noise accumulation and insufficient cross-modal feature utilization due to cascaded structures. To address these issues, this study proposed a coarse-to-fine registration and fusion network (CoReFuNet) that integrated a one-stage registration module with a cross-modal fusion module for multimodal typhoon cloud image classification. The registration module adopted a one-stage coarse-to-fine strategy, using cross-modal style alignment to address radiometric difference and global spatial registration by affine transformations to resolve positional differences. Bidirectional local feature refinement (BLFR) then ensured precise adjustments, facilitating fine registration by evaluating feature points in each image. Following registration, the fusion module employed a dual-branch alternating enhancement (DAE) approach, which reduced noise by learning cross-modal mapping relationships and applying feedback-based adjustments. Additionally, a cross-modal feature interaction (CMFI) module merged low-level, high-level, intra-modal, and intermodal features through a residual structure, minimizing modality differences and maximizing feature utilization. Experiments on the FY-HMW (Feng Yun-Himawari) dataset, constructed using data from the Feng Yun and Himawari satellites, showed that the CoReFuNet outperformed existing registration methods (VoxelMorph and SIFT) and fusion methods (IFCNN and DenseFuse), achieving 84.34% accuracy and 87.16% G-mean on the FY test dataset, and 82.88% accuracy and 85.54% G-mean on the HMW test dataset. These results showed significant improvements, particularly in unaligned data scenarios, highlighting the potential for real-world typhoon monitoring and forecasting.