AI Agent Readiness Across 100,000 Websites

This study evaluates web readiness for autonomous AI agents across the top 100,000 domains in the Tranco global ranking using 120 deterministic technical indicators across six operational layers. While direct requests retrieve readable HTML on nearly half of all websites (48.66%), dedicated machine interfaces such as llms.txt (7.16%), OpenAPI schemas (0.23%), and Model Context Protocol servers (0.05%) remain rare. Domains satisfy an average of 21.33% of evaluated criteria when counting verified capabilities, reflecting substantial operational uncertainty caused by live-web network drops, timeouts, and bot defenses. These empirical observations establish a baseline for web agent infrastructure, indicating that autonomous systems must continue relying on browser rendering, DOM heuristics, and fallback scraping until structured machine contracts achieve widespread adoption.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-10
DOI
https://doi.org/10.5281/zenodo.24121070
Primary Topic
Web Data Mining and Analysis
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

AI Agent Readiness Across 100,000 Websites

Vedang Ratan Vatsa
Zenodo (CERN European Organization for Nuclear Research)
Web Data Mining and Analysis
article

AI Agent Readiness Across 100,000 Websites

Vedang Ratan Vatsa
article en

Abstract

This study evaluates web readiness for autonomous AI agents across the top 100,000 domains in the Tranco global ranking using 120 deterministic technical indicators across six operational layers. While direct requests retrieve readable HTML on nearly half of all websites (48.66%), dedicated machine interfaces such as llms.txt (7.16%), OpenAPI schemas (0.23%), and Model Context Protocol servers (0.05%) remain rare. Domains satisfy an average of 21.33% of evaluated criteria when counting verified capabilities, reflecting substantial operational uncertainty caused by live-web network drops, timeouts, and bot defenses. These empirical observations establish a baseline for web agent infrastructure, indicating that autonomous systems must continue relying on browser rendering, DOM heuristics, and fallback scraping until structured machine contracts achieve widespread adoption.

Zenodo (CERN European Organization for Nuclear Research)
Web Data Mining and Analysis
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon