Image Diversity Evaluation Metrics Correlated with Human Subjectivity and Prediction of Image Diversity in Text-to-image Synthesis

This study investigates the diversity of image sets generated from textual descriptions by Text-to-image (T2I) models. Understanding this diversity is crucial for evaluating the relationship between textual inputs and the resultant images, which could significantly streamline the analysis of T2I models and minimize the human effort involved in generating desired images. Yet, this exploration raises two primary research questions: Firstly, the most appropriate metric for evaluating diversity, in terms of its correlation with human subjective assessments, remains unidentified. Secondly, the capability to predict the diversity of images produced by T2I models―encompassing both the feasibility and precision of such predictions―is yet to be established. To address these inquiries, our research was structured around two experiments. The first experiment aimed to identify the diversity evaluation metric that most closely aligns with human judgment by comparing the correlations between several metrics and human evaluations. The second experiment focused on developing a model capable of predicting the diversity of generated image sets, testing both its practicality and the accuracy of its predictions. Our findings reveal that the Vendi Score exhibits the highest correlation with human subjective evaluations, suggesting its effectiveness as a diversity metric. Additionally, while constructing a model to predict diversity proved feasible, the accuracy of these predictions was found to be significantly influenced by the specific objects described in the input texts. This highlights the nuanced relationship between textual descriptions and the diversity of generated images, suggesting avenues for further research to enhance the predictive model’s accuracy.