Tag-First: Mitigating Distributional Bias in Synthetic User Profiles through Controlled Attribute Generation

Addressing the critical need for robust bias testing in AI systems, current methods often rely on overly simplistic or rigid persona templates, limiting the depth and realism of fairness evaluations.We introduce a novel framework and an associated tool designed to generate high-quality, diverse, and configurable personas specifically for nuanced bias assessment.Our core innovation lies in a two-stage process: first, generating structured persona tags based solely on user-defined configurations (specified manually or via an included agent tool), ensuring attribute distributions are controlled and crucially, are not skewed by an LLM's inherent biases regarding attribute correlations during the selection phase.Second, transforming these controlled tags into various realistic outputs-including natural language descriptions, CVs, or profiles-suitable for diverse bias testing scenarios.This tag-centric approach preserves ground-truth attributes for analyzing correlations and biases within the generated population and downstream AI applications.We demonstrate the system's efficacy by generating and validating 1,000 personas, analyzing both the adherence of natural language descriptions to the source tags and the potential biases introduced by the LLM during the transformation step.The provided dataset, including both generated personas and their source tags, enables detailed analysis.This work offers a significant step towards more reliable, controllable, and representative fairness testing in AI development.