Human-Centered AI for Discovering Student Engagement Profiles on Large-Scale Educational Assessments

Large-scale assessments play a key role in education: they provide insights for educators and stakeholders about what students know and are able to do, which can inform educational policies and interventions. Besides overall performance scores and subscores, educators need to know how and why students performed at certain proficiency levels to improve learning. Process/log data contain nuanced information about how students engaged with and acted on tasks in an assessment, which hold promise of contextualizing a performance score. However, one isolated action event observed in process data may be open to multiple interpretations. To address this challenge, in the current study, we propose to integrate sequential process data with response data to create engagement profiles to better reflect students' test-taking processes. Most importantly, we propose to use AI algorithms to assist and amplify human expertise in the creation of students’ engagement profiles, so that the information extraction from the multi-source (performance and process) data can be scaled up to enhance the value of large-scale assessments in teaching and learning. We leveraged various machine learning techniques and developed a general framework of the human-centered AI approach to help human experts efficiently and effectively make sense of the multi-source data. Using a mathematics item block from the National Assessment of Educational Progress (NAEP) for illustrations, data from over 14,000 students resulted in ten preliminary profiles, more than half of which were associated with low performing students. These engagement profiles are expected to generate rich and meaningful feedback for educators and stakeholders.