Prompt Selection, Denominators and Repeat Noise in AI Citation Tracking: A Recount of Saved Scraper Runs
Reports of AI visibility often present a citation rate as a trend line. This paper asks how much of a reported movement can come from the counting specification and repetition noise rather than changed answers. It recounts saved runs from one project: 48 informational prompts, three repetitions each, collected twice on 1 November 2026 through a ChatGPT scraper and a Gemini scraper under identical settings. Twenty domain and scraper pairs, the ten most cited domains per scraper in the first sweep, are scored under ten counting rules, and moves are judged against relabeling nulls. For one domain, the repeat sweep gave 27 of 144 runs (18.75%) over all prompts and 23 of 45 runs (51.11%) over the prompts that had cited it in the first sweep. These are run-presence rates on different prompt populations. A share of 27 of 647 citation slots (4.17%) uses a different unit and is not a comparable rate. Across the 20 pairs the selected-prompt rate was a median 4.0 times the default rate (range 2.1 to 12.0). Under the specified selection-only null, 18 of 20 observed selected-prompt rates fell within the central 95% simulation range; this is a compatibility result under an exchangeability assumption, not evidence that selection caused the difference. The direction of the change between the two sweeps depended on the counting rule for 10 of 20 pairs. The largest of ten selected moves reached 7 runs in 52% to 63% of relabelings, against 0% to 26% for a single domain chosen in advance. Between 49% and 68% of the variance in a domain's citation outcome (median over domains) lay between repetitions within a sweep, and 0% to 4% lay between sweeps. The records support a methods demonstration, not a population estimate: one date, one repeat, two scrapers and one project's prompts. The paper closes with a reporting checklist derived from these results.
Authors
- Ankur Shrestha (ORCID: https://orcid.org/0009-0009-6036-5849)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.5281/zenodo.24163693
- Primary Topic
- scientometrics and bibliometrics research
- Type
- preprint