From Raw LLMs to Calibrated Silicon Samples: An Empirical Validation and Methodological Debiasing Against the Israeli Voice Index

Background and Objectives: Public opinion research faces an acute structural crisis characterized by plummeting response rates, escalating fieldwork costs, and nonresponse bias across polarized demographic strata. Within computational social science, "silicon sampling" (conditioning large language models on socio-demographic personas to simulate survey respondents) has emerged as a promising paradigm. However, existing validation benchmarks frequently rely on aggregate correlations (r = 0.65 to 0.78) that mask substantial intra-question distributional error, while exhibiting severe roleplay decisiveness, mode collapse in international relations, and ideological over-uniformity. This study presents a two-wave empirical validation and methodological debiasing of SekerAI, benchmarked against the September 2026 Israeli Voice Index (N = 751) by the Israel Democracy Institute (IDI).Methodology: We conducted a controlled two-wave experiment across 24 national questions (108 discrete response categories). In Wave 1 (Uncalibrated Baseline, N = 751), synthetic agents generated via census-stratified probability vectors answered standard questionnaire items. In Wave 2 (Calibrated Simulation, N = 751), we engineered four structural debiasing mechanisms: (1) an Epistemic Humility Gate conditioning option selection on news salience and cognitive hesitation; (2) a Subjective Citizen Lens replacing textbook foreign policy with grassroots security skepticism; (3) Relevance-Gated Political-Ideological Anchoring paired with an Economic Pressure Index (1-10) and behavioral decision vectors; and (4) Anti-Hedging Scale Dispersion to prevent artificial mode collapse. Evaluations measured intra-question Total Variation Distance (TVD), Mean Absolute Error (MAE), and Pearson correlation (r) against a Uniform Null Baseline.Empirical Findings: Wave 1 exhibited strong aggregate correlation (option-level r = 0.804, bloc-level r = 0.862) but suffered from classical LLM distortions: mean TVD of 30.7%, MAE of 13.3 percentage points (pp), and severe "Don't know" under-representation (0.7% vs. 9.3% human mean). Wave 2 improved accuracy across 21 of 24 questions (87.5%): mean TVD dropped by 5.8 pp to 24.9% (a 19% relative error reduction), MAE shrank to 10.9 pp, option-level correlation rose to r = 0.844 (bloc-level r = 0.879), and "Don't know" selection reached 8.7% (within 0.6 pp of humans). Notably, TVD fell by 10.9 pp on international relations (48.2% to 37.3%) and was nearly halved on October 7 intelligence disclosures (20.5% to 10.9%).Conclusion: Raw LLMs cannot serve as faithful synthetic polls out of the box due to systematic decisiveness distortions. However, when grounded in formal survey methodology, calibrated silicon sampling achieves marked distributional convergence toward human benchmarks, offering a transparent instrument for pre-fieldwork diagnostics within the Total Survey Error (TSE) framework.

Authors

Publication Details

Journal
SocArXiv (OSF Preprints)
Published
2026-11-13
DOI
https://doi.org/10.31235/osf.io/v68zd
Primary Topic
Survey Methodology and Nonresponse
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

From Raw LLMs to Calibrated Silicon Samples: An Empirical Validation and Methodological Debiasing Against the Israeli Voice Index

Dor Posner
2 links
SocArXiv (OSF Preprints)
Survey Methodology and Nonresponse
preprint

From Raw LLMs to Calibrated Silicon Samples: An Empirical Validation and Methodological Debiasing Against the Israeli Voice Index

Dor Posner
preprint en

Abstract

Background and Objectives: Public opinion research faces an acute structural crisis characterized by plummeting response rates, escalating fieldwork costs, and nonresponse bias across polarized demographic strata. Within computational social science, "silicon sampling" (conditioning large language models on socio-demographic personas to simulate survey respondents) has emerged as a promising paradigm. However, existing validation benchmarks frequently rely on aggregate correlations (r = 0.65 to 0.78) that mask substantial intra-question distributional error, while exhibiting severe roleplay decisiveness, mode collapse in international relations, and ideological over-uniformity. This study presents a two-wave empirical validation and methodological debiasing of SekerAI, benchmarked against the September 2026 Israeli Voice Index (N = 751) by the Israel Democracy Institute (IDI).Methodology: We conducted a controlled two-wave experiment across 24 national questions (108 discrete response categories). In Wave 1 (Uncalibrated Baseline, N = 751), synthetic agents generated via census-stratified probability vectors answered standard questionnaire items. In Wave 2 (Calibrated Simulation, N = 751), we engineered four structural debiasing mechanisms: (1) an Epistemic Humility Gate conditioning option selection on news salience and cognitive hesitation; (2) a Subjective Citizen Lens replacing textbook foreign policy with grassroots security skepticism; (3) Relevance-Gated Political-Ideological Anchoring paired with an Economic Pressure Index (1-10) and behavioral decision vectors; and (4) Anti-Hedging Scale Dispersion to prevent artificial mode collapse. Evaluations measured intra-question Total Variation Distance (TVD), Mean Absolute Error (MAE), and Pearson correlation (r) against a Uniform Null Baseline.Empirical Findings: Wave 1 exhibited strong aggregate correlation (option-level r = 0.804, bloc-level r = 0.862) but suffered from classical LLM distortions: mean TVD of 30.7%, MAE of 13.3 percentage points (pp), and severe "Don't know" under-representation (0.7% vs. 9.3% human mean). Wave 2 improved accuracy across 21 of 24 questions (87.5%): mean TVD dropped by 5.8 pp to 24.9% (a 19% relative error reduction), MAE shrank to 10.9 pp, option-level correlation rose to r = 0.844 (bloc-level r = 0.879), and "Don't know" selection reached 8.7% (within 0.6 pp of humans). Notably, TVD fell by 10.9 pp on international relations (48.2% to 37.3%) and was nearly halved on October 7 intelligence disclosures (20.5% to 10.9%).Conclusion: Raw LLMs cannot serve as faithful synthetic polls out of the box due to systematic decisiveness distortions. However, when grounded in formal survey methodology, calibrated silicon sampling achieves marked distributional convergence toward human benchmarks, offering a transparent instrument for pre-fieldwork diagnostics within the Total Survey Error (TSE) framework.

SocArXiv (OSF Preprints)
Peace, justice, and strong institutions
Survey Methodology and Nonresponse
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon