Revealing the Fairness Issues of Text-to-Image Generation Models Based on Metamorphic Testing

The Text-to-Image Generation(T2I) Models acquires implicit social biases during the training process, which can easily cause social disputes and negative impacts in sensitive fields such as news broadcasting, educational illustrations, and so on. There are many factors contributing to this situation. Most of the existing studies only focus on the combed biases of gender and skin color. Therefore, this paper proposes a method based on metamorphosis test to reveal the fairness of T2I models in terms of gender, skin color, and region. The part of speech and influential positions are divided in the input text, and the prompt text is modified by using names with regional characteristics in combination with three mutation operators: entity replacement, entity attribute enhancement, and multilingual description. The large language model is used for optimization to expand the test cases for generating images. To alleviate the test oracle problem of the T2I models, three metamorphic relations are constructed for verification in terms of gender, skin color, and image-text consistency. This paper has conducted a large number of experiments on T2I models: Kolors, Wan Xiang, and Stable Diffusion. The experimental results show that after variation using the variation operator, the proportion of female entities increases by at least 11.4%, and the proportion of dark-skinned objects increases by at least 15.7%.