Exploring Correlated Facial Attributes in Text-to-Image Models: Unintended Consequences in Synthetic Face Generation

Text-based face manipulation is a powerful tool for generating and editing facial attributes in synthetic data, particularly for addressing imbalances in biometric datasets and helping to build fair and privacy-aware systems. However, controlling specific attributes remains challenging, as modifying one attribute often leads to unintended changes in others, potentially introducing new biases. In this paper, we investigate the cause of these unintended changes, hypothesizing that they stem from attribute correlations in the training data. By analyzing three face attribute datasets (CelebA, MAAD-Face, FFText-HQ), we identify common patterns of attribute co-occurrence and assess the behavior of three state-of-the-art manipulation models (ManiCLIP, FFCLIP, and DeltaEdit) using attribute classifiers to detect unwanted modifications. We found two types of errors: one where more attributes were altered than intended, and another where changes failed to occur due to dependencies on the original attributes of the face. While we hypothesized that these manipulation issues might correlate with dataset attribute correlations, our analysis using the Matthews Correlation Coefficient found no statistically significant relationships, suggesting that other factors may contribute to these unintended changes. This work can serve as a starting point for further research into the underlying causes of attribute entanglement in face manipulation models, helping to improve understanding and address related challenges.