Metric for Evaluating Stable Diffusion Models Using Attention Maps

Stable diffusion models used for high-quality image generation have attracted increasing attention; thus, it is important to evaluate their quality. Synthetic image metrics are categorized into two types of measures: the first type focuses on image quality, and the second type focuses on text–image alignment. To evaluate text–image alignment, the new metric called the text–image alignment metric (TIAM) has been developed. The TIAM evaluates the alignment between the content specified in the prompt and the corresponding generated image based on the prompt template. The TIAM allows a comprehensive evaluation of the alignment between text prompts and images in terms of the type, number, and color of specified objects. To identify the specified objects, the TIAM employs the pretrained object detection and segmentation model YOLOv8. However, pretrained models cannot be applied to image classes or styles with which they have not been pretrained except when fine-tuning is applied. For nonprofessional users, fine-tuning a model or preparing a training dataset is difficult. In this study, we extend the TIAM to support various image classes and styles by incorporating attention maps acquired during the image generation process and a language-vision model (e.g., BLIP2). The experimental results demonstrate that the proposed method enables the TIAM to evaluate diverse images without requiring additional steps, such as fine-tuning.