Development and research of the efficiency of a website parsing information system using the Selenide framework

The article is devoted to the study of methods for automating data collection from websites using parsing technologies. The paper describes the main advantages of parsing compared to manual data collection, provides a classification of existing parsers, their capabilities, limitations and application in real projects. A detailed analysis of popular commercial and free parsers, such as Import.io, Webhose.io, Dexi.io, Scraperhub, ParseHub, Visual Scraper, Spinn3r, 80legs, Scraper, OutWit Hub, is carried out in order to determine their advantages and disadvantages in various usage scenarios. Particular attention is paid to the comparison of the Selenide and Selenide frameworks, which are widely used to automate interaction with web browsers. The conclusion is made about the feasibility of using the Selenide framework due to its simplified syntax, capabilities for working with dynamic content and support for intelligent waiting. The article presents the development of a custom parser based on Selenide, focused on the needs of small and medium-sized enterprises with a limited budget. The system is built on a modern technology stack, including Java 11, Python, PostgreSQL, Angular 12, Docker, Gradle, Kafka, Node.js. The program architecture, interaction between modules, and the relational database model for storing the obtained data are described in detail. The proposed approach allows you to configure the parser to work with different types of sites, provides high speed of information collection and processing, as well as flexibility in configuring sampling parameters. The created tool provides the opportunity to use containerization technologies to simplify the deployment and support of the application. The results of the work can be used to implement effective information search systems and automate routine data collection processes, which is especially important for companies that seek to optimize their business processes and reduce costs