Exploring GenAI-Powered Listening Test Development

The advent of Generative Artificial Intelligence (GenAI) has ushered in a transformative wave within the field of language education. However, the applications of GenAI are primarily in language teaching and learning, with assessment receiving much less attention. Drawing on task characteristics identified from a corpus of authentic prior tests, this study investigated the capacity of GenAI tools to develop a short College English Test-Band 4 (CET-4) listening test and examined the degree to which its content, concurrent, and face validity corresponded to those of an authentic, human-generated counterpart. The findings indicated that the GenAI-created test aligned well with the task characteristics of the target test domain, supporting its content validity, whereas sufficient robust evidence to substantiate its concurrent or face validity was limited. Overall, GenAI has demonstrated potential in developing listening tests; however, further optimization is needed to enhance their validity. Implications for language teaching, learning and assessment are therefore discussed.