Research on the Selection Method and Data Set Construction of Yi Language Stop Words

This paper aimed to complete the research on the selection method and data set construction of the Yi language stop words which has positive significance for information retrieval, text classification, sentiment analysis and online public opinion analysis. On the basis of word segmentation, data cleaning and word frequency statistics of Yi language text, used statistical methods such as Document Frequency, Term Frequency, and Entropy, analyzed and selected the top 100 high-frequency words that were ranked by these three measures, through the intersection of three sets of high-frequency words, 89 words were identified as potential stop words in the Yi language, and according to the characteristics of stop words, 20 function words were selected as stop words. Based on the methods of selecting stop words in Chinese, Mongolian, Tibetan, Uyghur, etc., completed the construction of the Yi language stop words dataset.