Exploring students’ AI literacy and its effects on their AI output quality, self-efficacy, and academic performance

Abstract Artificial intelligence (AI) technologies are advancing swiftly, both in terms of quantity and quality. Students must possess a solid understanding, abilities, and skills to successfully employ them. Therefore, the purpose of this quantitative study is to examine students’ AI literacy, including AI technological understanding, its practical application, critical appraisal, and its effects on students’ AI output quality, self-efficacy, and academic performance. Moreover, this research examines the effects of students’ AI self-efficacy and output quality on their academic achievement. An online survey was used to gather responses from 286 Austrian university students. Structural equation modeling was performed to test the research model. The findings indicate that AI technical understanding and the practical application significantly and positively influenced AI self-efficacy, while AI practical application had a significant and positive influence on both, AI self-efficacy and AI output quality. AI critical appraisal significantly and negatively impacted AI self-efficacy and AI output quality, whereas AI technical understanding, critical appraisal, practical application, self-efficacy, and output quality had an insignificant effect on students’ academic performance. The findings of this study may deepen the understanding of students’ AI literacy, contribute to the literature in this relatively new academic field, and help clarify the contradictory findings on the effects of AI-powered technology use on students’ learning outcomes. This research can also help integrate AI-powered tools into education effectively and improve students’ AI literacy so they can use them ethically and effectively. The results may also help revise and create new AI policies, curricula, and educational initiatives.