GLAT: The generative AI literacy assessment test

The rapid integration of generative artificial intelligence (GenAI) technology into education requires precise measurement of GenAI literacy to ensure that learners and educators possess the skills to engage with and critically evaluate this transformative technology effectively. Existing instruments often rely on self-reports, which may be biased. In this study, we present the GenAI Literacy Assessment Test (GLAT), a 20-item multiple-choice instrument developed following established procedures in psychological and educational measurement. Structural validity and reliability were confirmed with responses from 355 higher education students using classical test theory and item response theory, resulting in a reliable 2-parameter logistic (2PL) model (Cronbach's alpha = 0.80; omega total = 0.81) with a robust factor structure (RMSEA = 0.03; CFI = 0.97). Critically, GLAT scores were found to be significant predictors of learners' performance in GenAI-supported tasks, outperforming self-reported measures such as perceived ChatGPT proficiency and demonstrating external validity. These results suggest that GLAT offers a reliable and valid method for assessing GenAI literacy, with the potential to inform educational practices and policy decisions that aim to enhance learners' and educators' GenAI literacy, ultimately equipping them to navigate an AI-enhanced future. • The Generative AI Literacy Assessment Test (GLAT) is developed and validated as the first performance-based measure of generative AI (GenAI) literacy. • GLAT demonstrates strong validity and reliability in assessing GenAI literacy among higher education students. • GLAT scores effectively predict performance in GenAI tasks, outperforming self-reported ChatGPT proficiency. • The GLAT demonstrates the value of performance-based assessments in predicting real-world outcomes and informing educational practices.