How AI developers acquire training data: six channels and the shift away from the open web

Developers of large AI models obtain training data through six channels: collection from the open web without a licence, licences from publishers and platforms, commissioned human and expert work, the purchase or licence of organisations' internal records, purpose-built environments and synthetic data, and data from their own users and devices. This paper describes each channel from a public-record compilation read on 18 October 2026: 122 quantified supply facts, 111 disclosed licensing deals, 83 lawsuits, rulings and regulatory actions, 227 financial records of data vendors, 111 funding events, 110 brokerage records and 1,022 price observations, with a robots.txt census of 6,094 websites. The open web remains the largest source by volume, but access is narrowing: 11.4 percent of the census sites shut at least one AI crawler out of the whole site, and an earlier audit found restrictions on 28 percent or more of the most actively maintained sources of a standard web corpus. Of the 83 legal records, 70 concern content taken without a licence and 12 concern developers' use of their own users' data; none concerns the channels in which developers pay for data. Money is moving to those paid channels: disclosed licences, data vendors whose newest reported revenues or run rates exceed one billion dollars, and, since 2026, published programmes that buy organisations' records. We set out the scale, prices and legal exposure of each channel (not legal advice), the limits of the record, and what the shift changes for organisations that hold data. Keywords: artificial intelligence; training data; data governance; web crawling; robots.txt; data licensing; platform labour; privacy; copyright; synthetic data Use of AI: this paper was prepared with generative AI tools, which compiled and analysed the open datasets and drafted the text. The author takes responsibility for its content. Competing interests: the author is CEO of AIData.Broker, which acts for companies that license their data to AI developers and publishes the open datasets the paper draws on. Also at aidata.broker/research/paper-channels.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-18
DOI
https://doi.org/10.5281/zenodo.23452797
Primary Topic
Ethics and Social Impacts of AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

How AI developers acquire training data: six channels and the shift away from the open web

Yuma Heymans
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
article

How AI developers acquire training data: six channels and the shift away from the open web

Yuma Heymans
article en

Abstract

Developers of large AI models obtain training data through six channels: collection from the open web without a licence, licences from publishers and platforms, commissioned human and expert work, the purchase or licence of organisations' internal records, purpose-built environments and synthetic data, and data from their own users and devices. This paper describes each channel from a public-record compilation read on 18 October 2026: 122 quantified supply facts, 111 disclosed licensing deals, 83 lawsuits, rulings and regulatory actions, 227 financial records of data vendors, 111 funding events, 110 brokerage records and 1,022 price observations, with a robots.txt census of 6,094 websites. The open web remains the largest source by volume, but access is narrowing: 11.4 percent of the census sites shut at least one AI crawler out of the whole site, and an earlier audit found restrictions on 28 percent or more of the most actively maintained sources of a standard web corpus. Of the 83 legal records, 70 concern content taken without a licence and 12 concern developers' use of their own users' data; none concerns the channels in which developers pay for data. Money is moving to those paid channels: disclosed licences, data vendors whose newest reported revenues or run rates exceed one billion dollars, and, since 2026, published programmes that buy organisations' records. We set out the scale, prices and legal exposure of each channel (not legal advice), the limits of the record, and what the shift changes for organisations that hold data. Keywords: artificial intelligence; training data; data governance; web crawling; robots.txt; data licensing; platform labour; privacy; copyright; synthetic data Use of AI: this paper was prepared with generative AI tools, which compiled and analysed the open datasets and drafted the text. The author takes responsibility for its content. Competing interests: the author is CEO of AIData.Broker, which acts for companies that license their data to AI developers and publishes the open datasets the paper draws on. Also at aidata.broker/research/paper-channels.

Zenodo (CERN European Organization for Nuclear Research)
Openalex Percentile: Top 7%
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.