Faster and More Efficient Subject Image Generation for Text-to-Image Diffusion Models

In recent years, there has been significant progress in text-to-image generation models. However, text struggles to accurately describe abstract concepts like shapes and sizes. Some methods have been proposed to enhance text prompt by incorporating image prompts. While they have shown effective improvements, they either require substantial fine-tuning costs or struggle to effectively integrate text and image information. In our study, we delve into the issue of the difficulty in integrating text and image information in decoupled cross-attention and conduct visual analysis. We identify the presence of background-related tokens in image features as a key factor affecting text fidelity. To address this issue, we develop an algorithm to filter out these tokens. Additionally, we observe differences in the attention of Unet layers to text prompts and image prompts. Based on this finding, we optimize the flow of image information to reduce interference with text information. In summary, we introduce a new topic-customized method that requires no repeated training. It trains a plug-and-play image prompt adapter with only 417M parameters, lightweight yet powerful, surpassing existing models in both text and image consistency. Our code and pre-trained checkpoints will be available at https://github.com/YZBPXX/DDCA.