Assessment of the Reliability of Lexical Lists for Automated Evaluation of Proficiency in Russian as a Foreign Language

The assessment of language proficiency plays a crucial role in education, but it often relies on subjective evaluation methods, which can result in bias and inconsistencies in the results. To address this challenge, many researchers advocate for the use of automated and semi-automated assessment methods based on linguistic characteristics of texts. In this study, we explore the applicability of available vocabulary lists as tools for automatically evaluating the proficiency levels of students learning Russian. There exist several types of lexical lists, including frequency-based word lists and minimum vocabulary lists. In this research, we analyze four popular Russian-language lexical lists commonly employed in education and lexical knowledge analysis. We hypothesize that texts produced by students assessed at lower language proficiency level will predominantly contain frequently used words and low-level vocabulary items corresponding to the distribution of lexical elements by frequency or language proficiency level represented in these lists. Conversely, students assessed at a higher proficiency level are expected to employ less commonly used and more complex lexical units. By examining the correlation between these resources and student-generated texts, we aim to gain insight into the suitability of using lexical lists for evaluating proficiency in Russian. For the analysis of correlations between the selected lexical lists and student texts, we employ custom Python scripts. Additionally, we utilize a cluster analysis method known as Principal Component Analysis (PCA) to test the hypothesis that students at the same proficiency level tend to use a similar basic vocabulary with some degree of variation. This study raises important questions about the effectiveness of using lexical lists for assessing language proficiency. The findings may serve as a foundation for developing more accurate and comprehensive methods for evaluating lexical proficiency among students learning the Russian language.