The History and Labor of Web Scraping

Abstract Both the free labor of generating data online and the importance of data work which cleans, labels, and verifies these data are required for producing artificial intelligence (AI) systems. Between these two essential moments in AI production, we identify web scraping as the intercalary process of extracting data from the internet at scale. In this chapter, we highlight the essential contributions of web scraping to the production of machine learning training datasets and show that scraping is a distinct category of software labor whose understanding is important for the political economy of AI. The chapter characterizes the labor involved in both the production of scraping software and the labor required to conduct scraping operations. We provide a critical historical assessment of scraping dating back to the 1990s, tracing the development of scraping from a technical requirement for indexing the web in early communication software, to its commercialization and contemporary use in the development and production of large language models. We further describe the conditions of the labor of scraping through the analysis of contemporary documentation: industry-specific blogs, discussion boards, market reports, and conferences. We argue that the economic utility of data is dependent on its transformation from web data into datasets, and that this transformation is achieved through the labor of scraping. We suggest that the increased legal and technical barriers to the practice frame the current web scraping labor ecosystem, continuously demanding new kinds of human expertise.