Prompt Selection, Denominators and Repeat Noise in AI Citation Tracking: A Recount of Saved Scraper Runs

Reports of AI visibility often present a citation rate as a trend line. This paper asks how much of a reported movement can come from the counting specification and repetition noise rather than changed answers. It recounts saved runs from one project: 48 informational prompts, three repetitions each, collected twice on 1 November 2026 through a ChatGPT scraper and a Gemini scraper under identical settings. Twenty domain and scraper pairs, the ten most cited domains per scraper in the first sweep, are scored under ten counting rules, and moves are judged against relabeling nulls. For one domain, the repeat sweep gave 27 of 144 runs (18.75%) over all prompts and 23 of 45 runs (51.11%) over the prompts that had cited it in the first sweep. These are run-presence rates on different prompt populations. A share of 27 of 647 citation slots (4.17%) uses a different unit and is not a comparable rate. Across the 20 pairs the selected-prompt rate was a median 4.0 times the default rate (range 2.1 to 12.0). Under the specified selection-only null, 18 of 20 observed selected-prompt rates fell within the central 95% simulation range; this is a compatibility result under an exchangeability assumption, not evidence that selection caused the difference. The direction of the change between the two sweeps depended on the counting rule for 10 of 20 pairs. The largest of ten selected moves reached 7 runs in 52% to 63% of relabelings, against 0% to 26% for a single domain chosen in advance. Between 49% and 68% of the variance in a domain's citation outcome (median over domains) lay between repetitions within a sweep, and 0% to 4% lay between sweeps. The records support a methods demonstration, not a population estimate: one date, one repeat, two scrapers and one project's prompts. The paper closes with a reporting checklist derived from these results.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24163693
Primary Topic
scientometrics and bibliometrics research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Prompt Selection, Denominators and Repeat Noise in AI Citation Tracking: A Recount of Saved Scraper Runs

Ankur Shrestha
Zenodo (CERN European Organization for Nuclear Research)
scientometrics and bibliometrics research
preprint

Prompt Selection, Denominators and Repeat Noise in AI Citation Tracking: A Recount of Saved Scraper Runs

Ankur Shrestha
preprint en

Abstract

Reports of AI visibility often present a citation rate as a trend line. This paper asks how much of a reported movement can come from the counting specification and repetition noise rather than changed answers. It recounts saved runs from one project: 48 informational prompts, three repetitions each, collected twice on 1 November 2026 through a ChatGPT scraper and a Gemini scraper under identical settings. Twenty domain and scraper pairs, the ten most cited domains per scraper in the first sweep, are scored under ten counting rules, and moves are judged against relabeling nulls. For one domain, the repeat sweep gave 27 of 144 runs (18.75%) over all prompts and 23 of 45 runs (51.11%) over the prompts that had cited it in the first sweep. These are run-presence rates on different prompt populations. A share of 27 of 647 citation slots (4.17%) uses a different unit and is not a comparable rate. Across the 20 pairs the selected-prompt rate was a median 4.0 times the default rate (range 2.1 to 12.0). Under the specified selection-only null, 18 of 20 observed selected-prompt rates fell within the central 95% simulation range; this is a compatibility result under an exchangeability assumption, not evidence that selection caused the difference. The direction of the change between the two sweeps depended on the counting rule for 10 of 20 pairs. The largest of ten selected moves reached 7 runs in 52% to 63% of relabelings, against 0% to 26% for a single domain chosen in advance. Between 49% and 68% of the variance in a domain's citation outcome (median over domains) lay between repetitions within a sweep, and 0% to 4% lay between sweeps. The records support a methods demonstration, not a population estimate: one date, one repeat, two scrapers and one project's prompts. The paper closes with a reporting checklist derived from these results.

Zenodo (CERN European Organization for Nuclear Research)
scientometrics and bibliometrics research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon