Scrape this! A benchmark application for evaluating web scrapers and bot detection techniques

Web scraping is a popular technique used for gathering data from the internet. However, some website owners feel the need to protect their data from such scrapers. This paper looks at the current research on the topic of web scraping and bot detection, and identifies an issue with testing and comparison of different scraper configurations and bot detections techniques. A benchmark application is created through a design study to address the found issues. This benchmark application is designed to run scraper configurations against bot detection techniques on locally deployed websites that allows full reproducibility. To evaluate the application, a demonstration is made in the form of an experiment with the scraper library Crawlee and different web bot detectors. The results from the experiment show both that the Crawler library can avoid detection whereas a naive scraper implementation does not, and that there is a difference in detection between bot detector libraries. Finally, the paper contributes to the advancement of the research area by highlighting the existing problem with a lack of common benchmarks and a lack of fully published configurations and source code, and to provide a publicly available benchmark application that can be used and extended by the research community.