ColorViTAN: A hybridised approach using Vision Transformers and CycleGAN to add color to greyscale images

In this paper we introduce ColorViTGAN, an approach to image colorization that leverages the power of Vision Transformers (ViT) and the cyclic consistency of CycleGANs. While traditional colorization methods often struggle with semantic understanding and global consistency, our hybrid model addresses these challenges by combining the long-range dependency modelling of ViTs with the image-to-image translation capabilities of CycleGANs. Training our model is done on the diverse Places365 dataset and the CIFAR10 dataset, using the L channel of the Lab* color space as input and generating the corresponding a* and b* channels. The ViT-based generator captures intricate spatial relationships and contextual information, enabling more accurate and semantically consistent colorization. The discriminator, also ViT-based, ensures the generated colors are realistic and coherent. Our results show that ColorViTGAN outperforms state-of-the-art colorization methods in both qualitative human evaluations and quantitative metrics such as PSNR of 22.659, SSIM of 0.928 for Places365. The model shows particular strength in handling complex scenes with multiple objects and varying lighting conditions. Furthermore, the cyclic consistency loss of the CycleGAN component allows our model to learn from unpaired data, significantly expanding the potential training dataset.