Generation of Kazakhstan's unified national testing variants using AI: a platform for automatic task creation with expert control

This study examines the use of artificial intelligence for Automatic Item Generation (AIG) in the context of Kazakhstan's Unified National Testing (UNT) and presents a human-in-the-loop platform for scalable, expert-controlled test development. The objective is to evaluate whether large language models (LLMs) can reliably generate high-quality, isomorphic mathematics test items in the Kazakh language while preserving psychometric and pedagogical requirements. A hybrid AI system combining a local and a cloud-based LLM was implemented to perform semantic deconstruction of prototype items and constrained isomorphic generation of new variants. The pipeline included structured prompt engineering, parallel generation, and automated symbolic validation using Python and SymPy, followed by double-blind expert review. A stratified sample of 120 UNT mathematics items served as prototypes, from which 200 AI-generated clones were produced and validated. Six qualified subject-matter experts conducted independent evaluations using standardized criteria. Inter-rater reliability reached a substantial level (Cohen's κ = 0.78). Results show that 97.5% of generated items were recommended for use after review, with 50.5% accepted without revision and 47.0% accepted after corrections. The most frequent revision needs involved difficulty calibration, wording clarity, and factual or curricular alignment. Expert interviews confirmed that AI generation significantly reduces development time but remains limited in higher-order cognitive item design and pedagogically grounded distractor construction, especially in a low-resource, morphologically complex language environment. The findings support a hybrid augmentation model in which AI accelerates large-scale item production while experts ensure linguistic, cultural, and psychometric validity. The proposed framework demonstrates practical potential for multilingual, high-stakes assessment systems and provides implementation guidelines for responsible AI integration in test development.