UAlign: LLM Alignment Benchmark for the Ukrainian Language

This paper introduces UAlign, the comprehensive benchmark for evaluating the alignment of Large Language Models (LLMs) in the Ukrainian language.The benchmark consists of two complementary components: a moral judgment dataset with 3,682 scenarios of varying ethical complexities and a dataset with 1,700 ethical situations presenting clear normative distinctions.Each element provides parallel English-Ukrainian text pairs, enabling crosslingual comparison.Unlike existing resources predominantly developed for high-resource languages, our benchmark addresses the critical need for evaluation resources in Ukrainian.The development process involved machine translation and linguistic validation using Ukrainian language models for grammatical error correction.Our cross-lingual evaluation of six LLMs confirmed the existence of a performance gap between alignment in Ukrainian and English while simultaneously providing valuable insights regarding the overall alignment capabilities of these models.The benchmark has been made publicly available to facilitate further research initiatives and enhance commercial applications.