Gradual Acceptability Judgments with LLMs: Evidence from Agreement Variation in Russian

This study examines the LLMs' performance on gradual acceptability judgments task.Previously, the linguistic competence of LLMs was evaluated using binary acceptability scales, which contradicts the theoretical concept of acceptability.We present a new benchmark KVaS (Korpus Variativnogo Soglasovanija 'Corpus of Variable Agreement') derived from syntactic experiments on variable agreement in Russian.Our dataset contains multiple phenomena of agreement variation, ideal for modeling diverse acceptability levels, and compiles 7013 sentences rated by native speakers on a 1-7 Likert scale.We evaluated two LLMs, mainly Russian-trained GigaChat-Pro and multilingual Mistral Large, comparing their capability to treat acceptability as a scale to the reference human scores from KVaS.We used prompting providing benchmark sentences in two modes: zero-shot mode included only instructions while a few-shot mode added training sentences and their scores.The results show that GigaChat-Pro underperformed compared to Mistral Large.GigaChat-Pro improved significantly in a few-shot mode while Mistral Large exhibited more stable behavior.The case study shows that Mistral can detect nearly all significant contrasts in an experiment, whereas GigaChat performed near-randomly.The corpus may be useful for ranking LLMs, fine-tuning, and enhancing Russian text generation quality.