Fetched, not followed: on one site, ClaudeBot and GPTBot repeatedly requested an llms.txt-format list and none of the 12 pages it listed

We planted 36 static files on one website, each with exactly one intended route, and recorded which verified AI and search crawlers fetched them. Twelve pages were listed only inside an llms.txt-format file, announced only by its own robots.txt Sitemap line. Fetched, not followed. In the preregistered 21-day window, verified ClaudeBot requested the list 234 times and GPTBot 21 times. Neither fetched any of the 12 listed pages. Through 2026-11-17 the totals are 269 and 34, still 0 of 12. Other routes worked. The same crawlers fetched every file announced by its own robots.txt Sitemap line, and ClaudeBot fetched every file linked from the site footer. Google. Googlebot and GoogleOther followed a plain HTML link to a text file within hours. No Googlebot request was observed in 28 days to six plain text notes announced only by Sitemap lines (the notes were not valid sitemaps). Forecasts. Two of five preregistered predictions met their thresholds. What this is and is not. One site, one route for the list, a fetch log. It shows retrieval, not use, and it does not test llms.txt generally or the llms.txt filename. Two leading explanations remain, and the next design is built to separate them. Rival explanations are weighed with Heuer's Analysis of Competing Hypotheses. The preregistration was an internal document; its provenance, a mid-window logger change, the lost raw logs and the late reads are disclosed in full. Data. The deposit holds every logged crawler request to the study folder with classifier verdicts, and a script that re-derives every number in the note. The 12 control file names and the preregistration are withheld until after the 90-day re-read on 2027-01-12, to keep the control intact. Their SHA-256 hashes are in the deposit, and both files will be added as a new version of this record. Follows Hall (2026), Discovery, not adoption, doi:10.5281/zenodo.22814843.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24325558
Primary Topic
Web Data Mining and Analysis
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Fetched, not followed: on one site, ClaudeBot and GPTBot repeatedly requested an llms.txt-format list and none of the 12 pages it listed

Nathan Hall
2 links
Zenodo (CERN European Organization for Nuclear Research)
Web Data Mining and Analysis
preprint

Fetched, not followed: on one site, ClaudeBot and GPTBot repeatedly requested an llms.txt-format list and none of the 12 pages it listed

Nathan Hall
preprint en

Abstract

We planted 36 static files on one website, each with exactly one intended route, and recorded which verified AI and search crawlers fetched them. Twelve pages were listed only inside an llms.txt-format file, announced only by its own robots.txt Sitemap line. Fetched, not followed. In the preregistered 21-day window, verified ClaudeBot requested the list 234 times and GPTBot 21 times. Neither fetched any of the 12 listed pages. Through 2026-11-17 the totals are 269 and 34, still 0 of 12. Other routes worked. The same crawlers fetched every file announced by its own robots.txt Sitemap line, and ClaudeBot fetched every file linked from the site footer. Google. Googlebot and GoogleOther followed a plain HTML link to a text file within hours. No Googlebot request was observed in 28 days to six plain text notes announced only by Sitemap lines (the notes were not valid sitemaps). Forecasts. Two of five preregistered predictions met their thresholds. What this is and is not. One site, one route for the list, a fetch log. It shows retrieval, not use, and it does not test llms.txt generally or the llms.txt filename. Two leading explanations remain, and the next design is built to separate them. Rival explanations are weighed with Heuer's Analysis of Competing Hypotheses. The preregistration was an internal document; its provenance, a mid-window logger change, the lost raw logs and the late reads are disclosed in full. Data. The deposit holds every logged crawler request to the study folder with classifier verdicts, and a script that re-derives every number in the note. The 12 control file names and the preregistration are withheld until after the 90-day re-read on 2027-01-12, to keep the control intact. Their SHA-256 hashes are in the deposit, and both files will be added as a new version of this record. Follows Hall (2026), Discovery, not adoption, doi:10.5281/zenodo.22814843.

Zenodo (CERN European Organization for Nuclear Research)
Web Data Mining and Analysis
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon