ColorGPT: Automatic Colorization with Generative Prompts and Transformer

Automatic image colorization, where grayscale images are transformed into plausible color images, is a critical task with applications in film restoration, digital archiving, and artistic creation. Despite progress, challenges persist in achieving realistic and contextually accurate colorization due to the ambiguity in mapping grayscale intensities to colors and the need for semantic understanding. We design ColorGPT, a framework integrating generative language models and advanced colorization techniques to address these challenges. Our approach features three key modules: (1) a Prompt Generation Module using large language models to create diverse and context-aware color descriptions, (2) a Score Estimation Module employing CLIP to rank and select the most semantically relevant descriptions, and (3) a Colorization Module with transformer for high-quality image generation. Experiments on Set5 and Set14 demonstrate that ColorGPT outperforms state-of-the-art methods, achieving PSNR improvements of 0.7 dB on Set5 and 1.2 dB on Set14.