Dynamic Cloud-based Search Engine: Puppeteer-Driven Crawler with TF-IDF for Accurate Ranking of Search Results

Internet comprises of billions of web pages, each offering different kind of information. Web search engines are useful in obtaining necessary data, facilitating processes based on different structures. These processes typically include crawling, indexing, searching and filtering or ranking of information. Paradoxically, data scaling is a challenge for effective information retrieval. Users often need to go through numerous pages to find the information they seek. Web crawlers are essential for search engines, as they constantly scan the web to identify new pages and detect ones that have been removed. Given rapid and ever-changing nature of web, managing and navigating through numerous URLs within web documents has become increasingly difficult. This paper presents a scalable, cloud-based search engine designed to efficiently crawl and index dynamic content from Single Page Applications (SPAs). By leveraging cloud computing, the system can manage large-scale web crawling tasks. It utilizes Puppeteer for site scraping and Cosmos DB for storing the collected data. To enhance efficiency, web scrapers operate concurrently, utilizing Chromium and Puppeteer Cluster to speed up crawling and indexing, particularly on websites with extensive JavaScript content. This search engine addresses limitations of traditional crawlers which involve dynamic content, ultimately enhancing search performance and accuracy. Additionally, integration of Artificial Intelligence and Machine Learning (AIML) techniques further optimize indexing and retrieval process, thus improving system's overall efficiency. Designed for scalability and reliability, the search engine is capable of handling large volumes of web data while remaining responsive.