CoupletEval: A Novel Benchmark for Assessing Chinese Linguistic Proficiency in Large Language Models

The evaluation of linguistic capabilities in large language models (LLMs) has long lacked rigorous, unified methodologies that address both intrinsic language properties and human interpretability. To bridge this gap, we propose a novel framework that leverages Chinese couplets—a traditional literary form with strict prosodic and semantic constraints—as a challenging benchmark. Our methodology introduces both formal metrics (length match, tonal pattern opposition, part-of-speech correspondence, structural alignment, and character non-repetition) and content metrics (rhythmic harmony, semantic relevance, and imagery correspondence), enabling multidimensional assessment. A key innovation is the integration of a model self-evaluation mechanism, allowing LLMs to assess their own outputs and providing valuable insights into their metacognitive abilities. Systematic experiments with contemporary models—including Qwen3, Gemma3, DeepSeek-R1, and DeepCoder—across various parameter scales (0.6B to 7B) empirically validate the proposed approach. Results show strong concordance between model self-evaluation and conventional computational metrics, particularly in scenarios requiring deep cultural contextualization and strict formal constraints. These methodological advancements not only support computational model optimization but also promote the development of culturally informed artificial intelligence, establishing Chinese couplets as a robust benchmark for evaluating advanced language understanding and generation capabilities.