Image restoration employing cross-ViT combined generative adversarial networks

We propose Cross-ViT GAN, a novel image restoration model that addresses the limitations of CNNs by integrating Cross-ViT into GANs. The Cross-ViT GAN includes both generators and discriminators. In the generator, we replace the original normalization layer with a self-modulation layer, allowing the intermediate feature map to adapt to the input noise vector during backpropagation, thus enhancing generated image quality. The discriminator incorporates Cross-ViT, dividing the input image into large and small branches and using cross-attention to exchange features between different scales, enhancing image features for better recognition. We use three datasets, CelebA, Palces2 and CIFAR-10, to evaluate the performance of the model. The experimental results show that the evaluation metrics PSNR:26.48, SSIM:0.923, IS:4.11, and FID:1.65 of the Cross-ViT GAN model in the CelebA dataset outperform those of other comparative models, which proves the effectiveness of the Cross-ViT GAN model.