Prediction Is Not Tracking: Separating Predictive Skill from Temporal Information in Longitudinal Synthetic Surveys
Large language models are increasingly proposed as synthetic survey respondents, but the evidence is cross-sectional — resemblance at a point in time. The harder question is whether a model reproduces change. We formalise it for longitudinal synthetic surveys and show that a screen stopping at persistence cannot answer it. Two estimands follow: skill against persistence, and increment over an admissible family containing a mean-reversion baseline. In simulation, a country-mean predictor beats persistence by 38–41% on weakly persistent targets while receiving no period-specific input, so a persistence screen admits an object that cannot know the answer. Applied to a monthly 30-country consumer survey over ten held-out years, both open-weight models fail on every arm: none beats persistence (SS_persist −0.62 to −7.16) or adds information to the training-selected baseline family (SS_increment within ±0.09). The arm ranked first by persistence alone is nearly calibrated and carries no within-country information (r = −0.03). Between-country correlations of 0.58–0.85 with history coexist with within-country correlations never above 0.27. A preregistered test on the political item with the most tracking room (2.8 times the variance share) replicates this: every registered threshold holds but one direction did not; within-country correlations stay below 0.30.
Authors
- Yuteng Li (ORCID: https://orcid.org/0009-0003-1809-6477)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24400581
- Primary Topic
- Survey Methodology and Nonresponse
- Type
- preprint