How Reproducible Is AI Search Visibility? Prompt-Paraphrase Sensitivity and a Confidence-Aware Measurement Protocol for Generative Engine Optimization

Generative Engine Optimization (GEO) is now sold as a measurable service, yet its standard measurement is a single point estimate: run a fixed prompt basket through a model, count brand mentions, report a percentage. We ask whether that number is reproducible, using three controlled experiments on Russian-language queries. (A) Across four dates, the mean pairwise Jaccard overlap of cited domains was 0.356 (33 queries, openai:gpt-4o-mini-search-preview). (B) Across five back-to-back runs within ten minutes, it was 0.722 (8 queries, perplexity:sonar); only 49 of 95 query-domain pairs (51.6%) survived all five runs. (C) Our main result isolates a variance source the market ignores: prompt phrasing. In an 8 intents x 5 paraphrases x 5 repeats design (200 answers), we define a Prompt-Paraphrase Index PPI = W - B, with W the within-phrasing reproducibility and B the between-phrasing similarity. We measured W = 0.723, B = 0.431, PPI = 0.292 (cluster bootstrap 95% CI [0.206, 0.372]; permutation test, Holm-corrected max p = 0.0008). Of 300 intent-domain pairs, 173 (58%) had a presence range of at least 50 percentage points across paraphrases, and for 35% at least one phrasing cited a domain in all five runs while another cited it in none. The vendor who writes the prompt basket controls this variance, so an unaudited GEO percentage is not a stable property of a website. We then propose the GEO-Signal Passport, operational metrics that replace the single percentage: Confidence-Adjusted Visibility (a Jeffreys lower bound), prompt-basket decomposition, a three-axis stability report, citation half-life, a citation/absorption/fidelity split, and a difference-in-differences design for attribution. All formulas are numerically verified; code and data are released for replication.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-07-22
DOI
https://doi.org/10.5281/zenodo.21498261
Primary Topic
Advanced Multi-Objective Optimization Algorithms
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

How Reproducible Is AI Search Visibility? Prompt-Paraphrase Sensitivity and a Confidence-Aware Measurement Protocol for Generative Engine Optimization

Stanislav Kirichenko
Zenodo (CERN European Organization for Nuclear Research)
Advanced Multi-Objective Optimization Algorithms
preprint

How Reproducible Is AI Search Visibility? Prompt-Paraphrase Sensitivity and a Confidence-Aware Measurement Protocol for Generative Engine Optimization

Stanislav Kirichenko
preprint en

Abstract

Generative Engine Optimization (GEO) is now sold as a measurable service, yet its standard measurement is a single point estimate: run a fixed prompt basket through a model, count brand mentions, report a percentage. We ask whether that number is reproducible, using three controlled experiments on Russian-language queries. (A) Across four dates, the mean pairwise Jaccard overlap of cited domains was 0.356 (33 queries, openai:gpt-4o-mini-search-preview). (B) Across five back-to-back runs within ten minutes, it was 0.722 (8 queries, perplexity:sonar); only 49 of 95 query-domain pairs (51.6%) survived all five runs. (C) Our main result isolates a variance source the market ignores: prompt phrasing. In an 8 intents x 5 paraphrases x 5 repeats design (200 answers), we define a Prompt-Paraphrase Index PPI = W - B, with W the within-phrasing reproducibility and B the between-phrasing similarity. We measured W = 0.723, B = 0.431, PPI = 0.292 (cluster bootstrap 95% CI [0.206, 0.372]; permutation test, Holm-corrected max p = 0.0008). Of 300 intent-domain pairs, 173 (58%) had a presence range of at least 50 percentage points across paraphrases, and for 35% at least one phrasing cited a domain in all five runs while another cited it in none. The vendor who writes the prompt basket controls this variance, so an unaudited GEO percentage is not a stable property of a website. We then propose the GEO-Signal Passport, operational metrics that replace the single percentage: Confidence-Adjusted Visibility (a Jeffreys lower bound), prompt-basket decomposition, a three-axis stability report, citation half-life, a citation/absorption/fidelity split, and a difference-in-differences design for attribution. All formulas are numerically verified; code and data are released for replication.

Zenodo (CERN European Organization for Nuclear Research)
Advanced Multi-Objective Optimization Algorithms
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

How Reproducible Is AI Search Visibility? Prompt-Paraphrase Sensitivity and a Confidence-Aware Measurement Protocol for Generative Engine Optimization — Stanislav Kirichenko · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS