Crawler Algorithm of Network Public Opinion Topic Based on VSM

There are various forms of network information, and the sources of various network public opinion information are also different, and the amount of information involved is extremely large. The traditional public opinion collection work relies on manual completion, there are some problems such as the locality of monitoring objects and the lag time of discovery. Subject search engines target web pages related to a specific topic, weaken coverage, and provide more accurate and personalized search services. The topic crawler divides the topic related pages into blocks, establishes a hierarchical model, sorts the pages according to the relevance of the pages, and adds the links to the URL queue to provide the page text base for future retrieval. Based on the theme crawler architecture, this paper focuses on the research of Chinese word segmentation based on CRF and the theme crawler algorithm based on VSM, solves the key problems of the theme crawler technology for network public opinion, and can intelligently collect the web pages related to specific topics to avoid the access of unrelated web pages, providing a complete solution for the construction of the theme crawler system for network public opinion.