REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities

Recent advances in large language models (LLMs) have introduced the novel paradigm of using LLMs as judges, where an LLM evaluates and scores the outputs of another LLM, which often correlates highly with human preferences.However, the use of LLM-as-a-judge has been primarily studied in English.In this paper, we evaluate this framework in Russian by introducing the Russian Error tyPes Annotation dataset (REPA 1 ), a dataset of 1k user queries and 2k LLM-generated responses.Human annotators labeled each response pair expressing their preferences across ten specific error types, as well as selecting an overall preference.We rank six generative LLMs across the error types using three rating systems based on human preferences.We also evaluate responses using eight LLM judges in zero-shot and few-shot settings.We describe the results of analyzing the judges and position and length biases.Our findings reveal a notable gap between LLM judge performance in Russian and English.However, rankings based on human and LLM preferences show partial alignment, suggesting that while current LLM judges struggle with fine-grained evaluation in Russian, there is potential for improvement.Category # Queries Avg.# tokens in query Avg # tokens in response Example Generation 213 14.49 9.61 193.31 137.35 Rasskazhi istoriyu o tom, kak vazhno byt' dobrym i otzyvchivym.Tell a story about the importance of being kind and compassionate.Open QA 205 9.79 6.36 197.30 150.83 Opredelite i ob'yasnite znachenie muzykal'nogo termina allegro.Define and explain the meaning of the musical term allegro.Brainstorm 163 13.09 9.45 209.12 137.91 Sostav' spisok preimushchestv ispol'zovaniya solnechnykh paneley.Make a list of the advantages of using solar panels.Classify 120 30.63 18.98 158.46 136.50 Opredelite: ironiya, sarkazm ili yumor -Ty vsegda takoy umnyy, kogda spish'!Determine: irony, sarcasm, or humor -You're so smart when you sleep!.Rewrite 104 28.75 29.80 161.45 143.17 Perefraziruy s sinonimami: Vladelets magazina khochet bol'she pribyli i rosta!Rephrase using synonyms: The store owner wants more profit and growth.Extract 59 41.20 30.20 162.31 131.82 Razberite ukazannuyu datu na sootvetstvuyushchiye komponenty.<...> Break down the given date into its corresponding components.<...> Closed QA 49 91.16 67.80 145.48 152.08 O kakom vazhnom sobytii idet rech' v tekste?<...> What important event is being discussed in the text?<...> Chat 46 41.36 20.84 199.53 203.39 Predstav' chto ty otvechayesh' pyatiletnemu rebenku.Rasskazhi pro muzyku Shopena.Imagine you're answering a five-year-old.Tell them about Chopin's music.Error Type Example Request Following User query: Ob"yasni, pochemu lesa vazhny dlya zhizni na Zemle.LLM response: Original zapisi i kommentarii