Development and Comparative Analysis of Algorithms for Detecting Dialect Words of the Uzbek Language

This paper examines the issue of identifying dialect words in the Uzbek language and replacing them with formal equivalents. To solve the problem, the authors developed two algorithms, where the first algorithm is based on a rule-oriented (dictionary) approach. This algorithm analyzes the text word by word and, if necessary, applies morphological analysis to identify the root form and replace it with an equivalent of the literary language. The second algorithm is based on machine learning (logistic regression), where the model is trained on a dictionary of over 1000 lexemes, each of which has information about the regions of its use. To objectively evaluate each algorithm, two test datasets were formed (50 sentences each), differing in the number of dialect words in a sentence. Experiments showed that logistic regression outperformed the rule-oriented algorithm in all key metrics (Precision, Recall, F1-score), especially with many dialect words in one sentence. This result indicates a higher generalization ability of the machine learning model in recognizing non-trivial or rare shapes.