Leveraging a Vision Transformer and Adversarial Unsupervised Domain Adaptation to Generate the NIR Band on Out-of-Domain VHR RGB

The accessibility of very-high-resolution Remote Sensing (RS) near-infrared (NIR) imagery before 2010s is limited. In addition, public annotated image datasets often used as benchmarks contain only RGB information. Thus, RGB-to-NIR translation could greatly benefit the RS field. Generative adversarial networks (GANs) and more recently vision transformers (ViT) have shown promising results in generating spectrally super-resolved data. However, the RGB-to-hyperspectral studies scatter their attention to multiple bands, whereas the RGB-to-NIR studies have tackled only vegetation. In addition, prior research regarding NIR prediction on out-of-domain data is extremely limited, and attention models have not been previously tested in the RGB-to-NIR translation. Beyond that, unsupervised domain adaptation (UDA) has been overlooked in the enhancement of cross-domain band generation. This study, at first evaluates a conditional GAN and two attention networks on predicting NIR on out-of-domain RGB data. The properties of the out-of-domain data are those typically required in RS NIR prediction tasks (different regions/sensors/dates). In a following step, the study attempts to increase the source/target RGB radiometric similarity through CycleGAN-based UDA on unpaired bi-temporal data (lack of geographic correspondence in RGB source/target patches), and thus improve the initial NIR prediction. In the experiments, the vegetation, impervious and ground classes were assessed. It was shown that MST++ (ViT) produced the most satisfactory out-of-domain NIR predictions and that UDA through a CycleGAN version employing batch normalization managed to significantly enhance the NIR prediction when there was a substantial RGB radiometric domain gap.