How much does an AI answer change on its own? A small-sample measurement of generative-engine volatility

Generative search engines return different answers to the same question at different times, with no change to the underlying web or to the queried brand. This makes before-and-after measurement of any visibility work hard to trust. We ran a small, fixed basket of buyer questions on a clean ChatGPT surface across three non-consecutive windows, changing nothing in between, and recorded how often the set of named businesses changed. Across 36 observations the answer changed 27.8 percent of the time (10 status changes). We report this as a directional, small-sample result, not a population statistic. We set it alongside the large-scale study by Petra Labs (7,200 responses), which measured within-day model non-determinism and reached a compatible conclusion: short-run before-and-after comparisons are usually indistinguishable from sampling noise. Our contribution is narrow but distinct. We measure change across weeks rather than within a single day, which captures drift in the web and the index as well as model non-determinism, and which matches the window a real client engagement spans.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-07-26
DOI
https://doi.org/10.5281/zenodo.21606839
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

How much does an AI answer change on its own? A small-sample measurement of generative-engine volatility

David William
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Healthcare and Education
preprint

How much does an AI answer change on its own? A small-sample measurement of generative-engine volatility

David William
preprint en

Abstract

Generative search engines return different answers to the same question at different times, with no change to the underlying web or to the queried brand. This makes before-and-after measurement of any visibility work hard to trust. We ran a small, fixed basket of buyer questions on a clean ChatGPT surface across three non-consecutive windows, changing nothing in between, and recorded how often the set of named businesses changed. Across 36 observations the answer changed 27.8 percent of the time (10 status changes). We report this as a directional, small-sample result, not a population statistic. We set it alongside the large-scale study by Petra Labs (7,200 responses), which measured within-day model non-determinism and reached a compatible conclusion: short-run before-and-after comparisons are usually indistinguishable from sampling noise. Our contribution is narrow but distinct. We measure change across weeks rather than within a single day, which captures drift in the web and the index as well as model non-determinism, and which matches the window a real client engagement spans.

Zenodo (CERN European Organization for Nuclear Research)
Citigroup (US)
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.