The Datasets Crawling Based on Search Engine in Minor Fields AI Application

Datasets collection is the major problem to be solved for minor fields Artificial intelligence (AI) applications. Even though collecting data from internet with crawler technology is the most economic method, there still exist such difficulties as high dependence on host computer and network performance, and easily being blocked by firewalls. To address these difficulties, our ideas are as follows. Firstly, we will generate a request link containing keywords according to the search engine query interface and send a query request to the search engine through the browser. Secondly, by implanting JavaScript (JS) code to pass the verification, we use time delay processes to continuously send query requests to obtain all search results. Then the Uniform Resource Locator (URL) is extracted from the search results and placed in the URL list. Finally, the undownloaded links are randomly selected from the URL list, and a threshold value is set on the download amount of the target server to avoid being blocked by the firewall due to the frequent data downloads until all tasks are completed. In this paper we have two contributions: first, we use time delay processes in the algorithm to emulate the user's downloading behavior and avoid the identification of firewall by extending the downloading tasks; second, we sacrifice the time complexity of the algorithm to reduce its dependence on computer hardware and network. In this paper, the algorithm was deployed on a desktop computer to crawl data for 60 keywords, and the algorithm ran robustly for 5 hours to crawl a total of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{4 3, 0 0 0}$</tex> images, with a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{9 1. 8 \%}$</tex> match between image content and keywords.