Investigating the Utility of LLM-Based “Silicon Samples” in Political Science Conjoint Experiments

Abstract This research note is the first study to explore the viability of using LLM-generated respondents in forced-choice conjoint experiments. I generate naive “silicon samples” using three models, GPT 4o-Mini, Llama 3.2, and Magistral-small-2506, benchmarked to a large sample of live conjoint respondents, allowing for direct comparisons between human and silicon choices. The resulting analyses suggest that naively generated silicon samples are insufficient to simulate human decision-making in conjoint tasks; these results also suggest that there are systemic differences in the way demographic covariates shape LLM and human choices in experimental contexts. In total, this analysis makes two important contributions: it establishes a descriptive baseline understanding of LLM-generated respondent performance in conjoint experiments and suggests that naive silicon samples cannot accurately infer human choices using demographic traits in experimental decision making contexts. Finally, the implications of these results imply that LLMs consider covariates differently than live respondents do in micro-level decision making contexts.