Algorithm for Detection Relevant Text Elements Based on Morphological and Frequency Analysis

The main object of this work is to automate the process of detection key words and phrases using modern natural language processing methods, which will improve the structure and classification of text data, as well as adapt them for further integration with classification systems. For this purpose, algorithm for automatic detection of key words and phrases from texts in Russian language is proposed for use in working with complex multi-level classification systems such as UDC, GRNTI. This algorithm can work with single texts without linking them to collections of documents. А joint frequency and morphological analysis was used to detect keywords and phrases, take into account the structure of the document. When detection of key phrases, lexical and grammatical patterns consists of adjectives and nouns were used as well as stable combinations of nouns. The algorithm effective works with large texts divided into segments (ones of relevant). To adjust the rank of a relevant text element calculated using frequency analysis. А special coefficient is introduced that depend on the areas of occurrence of keywords. The comparative analysis showed that, in comparison with the TF-IDF and TextRank algorithms, the developed algorithm demonstrates high efficiency in detection key words. The integration of the automatic text analysis algorithm with classification systems discovers an additional opportunities to structure knowledge and to improve process efficiency the large amounts of data.