A web crawler method based on iframe supporting asynchronous requests

Economic analysis and decision-making are inseparable from economic data, and e-commerce platforms aggregate a vast amount of transaction data, user behavior data, and more. Web crawling technology is the primary technique for obtaining data from e-commerce websites. However, due to the impact of anti-crawling technology and website implementation technology, obstacles exist in the execution of crawlers. This paper comprehensively analyzes the characteristics of web crawling technology, anti-crawling technology, and asynchronous request technology on website pages. It suggests that the iframe tag's ability to automatically request and render web pages can be utilized by embedding the iframe tag within web pages. Then, JavaScript scripting can be employed to manipulate the DOM of the web page, parse the web page within the iframe, and obtain data. Verification through e-commerce platforms demonstrates that this method is indeed feasible, simple, and easy to use.