Adaptive Prompt Modifier for Text-to-Image Model

With the increasing prevalence of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), the quality of data generated by Large Models (LMs) is critically influenced by the design and formulation of prompts, especially for art generation prompts, which require even more meticulous design. However, due to the large search space of natural language, prompt engineering for LMs especially Text-to-Image(T2I) models is difficult. While manual prompts need to be done on a trial-and-error adhoc basis and lack diversity, prompts generated by LLMs fail to provide tailored prompts for the target type of synthetic data. To address these problems, we propose an adaptive prompt learning approach, AdaPTer, based on supervised fine-tuning on LLMs with a filtered subset of task model’s training set to supply domain knowledge for LLMs. Synthetic data guided by prompts generated by AdaPTer have better adaption to the target domain. This paper works on two main tasks with T2I models. Firstly, to find the correct direction for prompt optimization, we retrieve image-caption pairs from the training set of task T2I models. Due to the inherent correlation between text and image pairs in the training process of T2I models, image captions naturally emerge as highly effective prompts, subsequently offering a vast and generally reliable candidate prompt set for experimentation. By generating images with the T2I model, we explore the relationship between text and visual information in the pretrained T2I model to instruct prompt engineering. Secondly, we use similarity indicator to evaluate the quality of generated images, which represents quality of prompts from a consistency perspective, and prove the effectiveness of our prompt strategy. Experimental results demonstrate that AdaPTer exhibits superior performance compared to prompts refined directly using LLMs. Code is open in https://github.com/byeolnim11/AdaPTer.git.