Where human similarity holds and where it breaks: Evaluating the alignment of synthetic focus groups

Objective: the growing commercial use of large language model (LLM)-based synthetic participants raises questions about their validity. This study evaluates how reliably text-based synthetic focus groups reproduce human thematic content and interactional dynamics. Methods: the assessment used a matched dataset on animal product consumption and gender to generate 30 text-based synthetic focus-group sessions by manipulating agents’ profile levels (basic vs. enriched). Their transcripts were compared with those of five human focus groups using a novel three-level benchmark: thematic fidelity, interaction process, and speaker distinctiveness. Results: synthetic groups recovered a plausible subset of human themes but displayed excessive verbosity, internally resolved contrast, less contextually detailed discourse, and reduced speaker distinctiveness. Profile enrichment increased thematic coverage but concentrated discussions around shared ideas. Conclusions: these findings position synthetic focus groups as a promising tool for exploratory market research, while underscoring their limitations in capturing situated social interaction. Their suitability depends on the intended use, requiring multi-dimensional validation before application.