Improving T2I-Adapter via Integration of Visual and Textual Conditions with Attention Mechanism

Recently, relying solely on T2I has gradually proven insufficient to meet the demands for image generation. As a result, people have started exploring more controllable image-generation methods based on Diffusion technology. In addition to solely using textual descriptions to generate images, extra control conditions such as Sketch, Segmentation, and Canny are being considered. However, existing T2I control adapters mostly lack a comprehensive integration between textual and image conditions. Additionally, in multi-layer neural networks, certain features may be lost information. Therefore, this paper aims to address the aforementioned issues and propose a new architecture systematically. We utilize Cross Attention to merge textual descriptions and image features and introduce Coordinate Attention at each feature output to enhance the overall feature representation. Experimental results demonstrate that compares to state-of-the-art methods, this approach achieved superior evaluation metrics and exhibited visual effects more in line with human assessment.