Exploratory Investigation of Mathematics Engagement in Digitally Based Assessments

The NAEP 2017 Mathematics Assessment Process Data was the center of this exploratory investigation, examining sources of validity evidence for using process data indicators to obtain a snapshot of students’ engagement in mathematics. The COVID-19 pandemic and remote education requirements made it substantially more difficult for teachers to assess their students’ engagement in mathematics. As technology and access to it improve, students will receive more access to digital assessments and digital learning environments. It is important to probe how student behaviors in digital assessment and learning environments could indicate the quality of their engagement. After fulfilling the requirements through the Institute of Education Sciences (IES), access was granted to work with the NAEP 2017 Mathematics Assessment Process Data. The dataset included 2,800 eighth-grade students. These process data variables were analyzed: theme state, zoom state, text-to-speech, draw, erase, highlight, scratchwork, eliminate a choice, un-eliminate a choice, clear (or remove) answer, equation editor, and calculator functions. In addition, data regarding their mathematics proficiency were included. Validity evidence was collected based on the framework detailed in the Standards (AERA et al., 2014). The analyses used to gather validity evidence included the following: descriptive statistics, principal components analysis (PCA), latent profile analysis (LPA), correlations, regressions, and crosstabs analyses. The rate at which students utilized different features varied greatly. There were four potential engagement components per item, and the total number of potential engagement components across all items was eight. These components also varied to a great extent in their correlations with and predictions of mathematics proficiency. This study made clear that engagement is more complicated than once thought and that item type and available features could result in different types of engagement per item.