From Raw LLMs to Calibrated Silicon Samples: An Empirical Validation and Methodological Debiasing Against the Israeli Voice Index
Background and Objectives: Public opinion research faces an acute structural crisis characterized by plummeting response rates, escalating fieldwork costs, and nonresponse bias across polarized demographic strata. Within computational social science, "silicon sampling" (conditioning large language models on socio-demographic personas to simulate survey respondents) has emerged as a promising paradigm. However, existing validation benchmarks frequently rely on aggregate correlations (r = 0.65 to 0.78) that mask substantial intra-question distributional error, while exhibiting severe roleplay decisiveness, mode collapse in international relations, and ideological over-uniformity. This study presents a two-wave empirical validation and methodological debiasing of SekerAI, benchmarked against the September 2026 Israeli Voice Index (N = 751) by the Israel Democracy Institute (IDI).Methodology: We conducted a controlled two-wave experiment across 24 national questions (108 discrete response categories). In Wave 1 (Uncalibrated Baseline, N = 751), synthetic agents generated via census-stratified probability vectors answered standard questionnaire items. In Wave 2 (Calibrated Simulation, N = 751), we engineered four structural debiasing mechanisms: (1) an Epistemic Humility Gate conditioning option selection on news salience and cognitive hesitation; (2) a Subjective Citizen Lens replacing textbook foreign policy with grassroots security skepticism; (3) Relevance-Gated Political-Ideological Anchoring paired with an Economic Pressure Index (1-10) and behavioral decision vectors; and (4) Anti-Hedging Scale Dispersion to prevent artificial mode collapse. Evaluations measured intra-question Total Variation Distance (TVD), Mean Absolute Error (MAE), and Pearson correlation (r) against a Uniform Null Baseline.Empirical Findings: Wave 1 exhibited strong aggregate correlation (option-level r = 0.804, bloc-level r = 0.862) but suffered from classical LLM distortions: mean TVD of 30.7%, MAE of 13.3 percentage points (pp), and severe "Don't know" under-representation (0.7% vs. 9.3% human mean). Wave 2 improved accuracy across 21 of 24 questions (87.5%): mean TVD dropped by 5.8 pp to 24.9% (a 19% relative error reduction), MAE shrank to 10.9 pp, option-level correlation rose to r = 0.844 (bloc-level r = 0.879), and "Don't know" selection reached 8.7% (within 0.6 pp of humans). Notably, TVD fell by 10.9 pp on international relations (48.2% to 37.3%) and was nearly halved on October 7 intelligence disclosures (20.5% to 10.9%).Conclusion: Raw LLMs cannot serve as faithful synthetic polls out of the box due to systematic decisiveness distortions. However, when grounded in formal survey methodology, calibrated silicon sampling achieves marked distributional convergence toward human benchmarks, offering a transparent instrument for pre-fieldwork diagnostics within the Total Survey Error (TSE) framework.
Authors
- Dor Posner (ORCID: https://orcid.org/0009-0007-8872-3180)
Publication Details
- Journal
- SocArXiv (OSF Preprints)
- Published
- 2026-11-13
- DOI
- https://doi.org/10.31235/osf.io/v68zd_v1
- Primary Topic
- Survey Methodology and Nonresponse
- Type
- preprint