On Constructing Preference Pairs for Text-to-Image Generation

Aligning text-to-image models with human preferences remains a critical challenge despite rapid advances in diffusion-based generation. Direct Preference Optimization (DPO) offers an efficient fine-tuning paradigm by training on pairwise preference data rather than scalar rewards, but collecting highquality preference datasets at scale is costly. In this work, we present a practical framework for automated generation of DPO data in text-to-image settings. We introduce an ensemble scoring mechanism that leverages several pretrained evaluators to assign consolidated preference scores to model-generated samples and compare two pairing strategies: exhaustive enumeration and maximal margin selection. Through systematic experiments, we demonstrate that selecting only the most and least preferred images per prompt yields stronger supervision than using all pairs, and that sampling with forty to fifty diffusion steps captures the majority of alignment gains. We further show that assembling on the order of five thousand preference pairs is sufficient to achieve near-optimal performance. Our findings translate into concrete guidelines for data generation and model fine-tuning, enabling efficient alignment of text-to-image systems with user intent.