HVT-cGAN: Hybrid Vision Transformer cGAN for SAR-to-Optical Image Translation

Due to its capability for all-weather, all-time information acquisition, synthetic aperture radar (SAR) plays a vital role in the field of Earth observation. However, the specificity of the radar sensor and the complexity of electromagnetic scattering imaging physics result in SAR images lacking the intuitiveness of optical images, making them unsuitable for interpretation by nonexperts. A common approach to tackle this challenge is to use a conditional generative adversarial network (cGAN) to translate SAR images into optical images, thereby enhancing readability and assisting nonexperts in interpretation while filling the gaps in optical data due to acquisition constraints. Nevertheless, traditional cGAN-based methods are limited by inadequate global semantic information extraction and poor detail preservation, leading to translated images with incoherent texture and color, and blurred edge. To address these issues, we propose a hybrid vision transformer cGAN (HVT-cGAN) for SAR-to-optical image translation (S2OIT). In our proposed HVT-cGAN, the generator utilizes a convolutional stem for patch embedding and encoding. The parallel CNN branch and vision transformer (ViT) branch are employed for the extraction and mapping of local and global information, respectively. Moreover, we propose a novel attention-based feature fusion module, named the convolutional attention fusion module (CAFM), which can adaptively aggregate local and global information from parallel branches by learning both channel-wise and spatial-wise relations. Benefiting from these improvements, our method achieves superior performance in both qualitative and quantitative comparisons with other methods on the SEN1-2 dataset. In addition, the results of multiple ablation experiments validate the effectiveness of the proposed method.