A Pairwise Ranking-Based Method for Evaluating Cross-Corpus Difficulty Consistency

Text corpora for second language (L2) learners often suffer from inconsistencies in difficulty grading, stemming from variations in linguistic features, annotation standards, and contextual factors across datasets. To address this challenge, a pairwise ranking-based evaluation method is proposed, offering a systematic approach to rank text difficulty within and across corpora. Unlike traditional classification methods, the bert-based pairwise ranking model (BPRM) achieves superior performance in cross-prediction tasks, demonstrating strong ranking correlation metrics, such as NDCG and 1-JSD, across multiple datasets, including NES, RACE, and CLOTH. With average ranking correlations exceeding 0.9 in inter-corpus evaluations, BPRM outperforms baseline classification and neural models. This approach not only provides a fine-grained analysis of corpus consistency but also validates its applicability in transfer learning and zero-shot learning contexts, advancing both theoretical understanding and practical solutions for corpus difficulty evaluation.