The Role of Technical Knowledge in AI Literacy

As researchers investigate the role of AI in undergraduate education, questionnaires have become a prominent method of assessing AI literacy. These questionnaires tend to assume that STEM majors have higher AI literacy than non-STEM majors because of the technical aspects of their majors. However, students across disciplines report independent and voluntary use of AI in academics. Furthermore, few studies specifically examine the relationship between students' technical understanding of AI and their ability to effectively use AI. This study uses an embedded mixed-methods approach, with quantitative analysis of survey data supported by qualitative analysis of focus group interviews, to address this gap. Interviews revealed that students believe independent experimentation with AI allows them to discover how to effectively use AI tools. In support of this finding, Wilcoxon signed-rank tests showed a significant relationship between perceived understanding of AI and perceived ability to effectively use AI, but not between perceived technical knowledge and perceived ability to effectively use AI. The focus groups also revealed that students believe lacking a technical understanding of AI does not imply an individual isn't currently able to effectively use AI tools, and that students can gain an understanding of AI through experiential use. Additionally, the statistical tests of survey responses did not align with the results of the confirmatory factor analysis originally used to construct the survey. Student demographics and implications of technical knowledge in question phrasing likely impact question interpretation.