Integrating CLIP with Dynamic Memory Generative Adversarial Networks to Enhance Semantic Consistency in Text-to-Image Generation

This paper focuses on the challenge of a lack of strong semantic coherence between generated images and their corresponding text descriptions, as well as the low quality of initial image generation and limited detail capture in the Dynamic Memory Generative Adversarial Networks model. To tackle these issues, we propose the DM-CLGAN model, an enhanced variant of DM-GAN. By incorporating the CLIP model, DM-CLGAN achieves improved alignment between textual and image features. Furthermore, it introduces the Text-Image Affine Combination Module and the Convolutional Block Attention Module to optimize image details and enhance the quality of the initially generated images. In the second stage of the model, the dynamic memory network further refines the generated images. We utilized a quantitative assessment of the model's performance on the CUB-200-2011 dataset by employing key metrics such as Inception Score and Fréchet Inception Distance. The results of our study demonstrate that the generation quality exceeds that of other models, as confirmed by manual evaluation methods. Additionally, the model demonstrates exceptional generation capabilities on our custom-built ceramic wine bottle dataset, further validating the adaptability and generative efficacy of the proposed approach.