Learners’ priority criteria for evaluating data-driven learning tools: An analytic hierarchy process study with traditional and GenAI-based concordancers

Abstract This study investigates how language learners prioritize criteria for evaluating data-driven learning (DDL) tools and how these priorities relate to tool preference. While previous research has proposed multiple criteria for evaluating the perceived effectiveness of DDL tools in capturing the emic dimension of the learner experience, empirical work has scantily examined how learners differentially prioritize these criteria, or how such priority structures can be empirically incorporated into systematic comparisons of tools. Given the recent emergence of generative artificial intelligence (GenAI) in language learning, this study examines how GenAI-based tools may contribute to DDL when judged against learner-valued effectiveness criteria. Drawing on a literature review, six criteria were identified: sentence comprehensibility, relevance to semantic learning, relevance to syntactic learning, perceived pedagogical value, accessibility for independent use, and support for autonomous learning. Thirty-five Korean EFL university students completed an analytic hierarchy process (AHP) task. First, they made pairwise comparisons among the six criteria, yielding priority weights that indicated the relative importance learners attach to each aspect of DDL effectiveness. Second, the study compared the perceived effectiveness of a traditional online concordancer that accesses the British National Corpus with that of a custom-developed GenAI-based concordancer, prioritizing these weighted criteria. The results indicate that learners assign differentiated importance to the six criteria and that these priority patterns are associated with the concordance they regard as more effective. This study demonstrates the usefulness of AHP for modelling multi-criteria, perception-based evaluations of DDL tools and incorporating learners’ priority judgements into the comparison of alternative tool designs.