Advanced Literacies in the 21st Century: Examining A Narrative-Based Video Game as A Stealth Assessment for Comprehension

In our technology-rich 21st Century, the need for valid, reliable, and fair literacy assessments remains pertinent to the field of education. This study defines literacy as advanced literacy where comprehension is considered a holistic product of applying acquired knowledge, skills, or abilities (KSA) to contextualized situations. Conventional assessment methods have traditionally parceled out the construct into its component parts (i.e., vocabulary, phonics, fluency, etc.) as a means of understanding the meaning making process from a traditional literacy perspective. Stealth assessments utilize the Evidence-Centered Design (ECD) framework as a robust method for assessing complex constructs holistically within technology rich environments. Adapting from Shute’s (2011) stealth assessment design, this study operationalized video game behaviors as psychometrically sound measures of comprehension skills by employing McCreery’s et al. (2019) Behavioral Assessment Matrix (BAM).The Behavioral Assessment Matrix for Comprehension (BAM-COMP) was developed to systematically code operationally defined game-play behaviors that could be linked to comprehension. Rounds of expert scrutiny identified eight observable game-play behaviors that demonstrated high inter-observer agreement and acceptable internal consistency, confirming reliability of the instrument. Convergent validity for the BAM-COMP was established through small positive correlations with the well-established external literacy assessment The Woodcock- Johnson IV Tests of Achievement: Reading Cluster (WJ-IV: ACH Reading Cluster). T-tests showed that were significant differences in scores between the two instruments, indicating that practical applications of KSA may be distinctly different than rote knowledge for test taking procedures. Concurrent validity was established through multiple regression analyses by regressing the WJ-IV: ACH Reading Cluster onto the BAM-COMP subscales. The results indicated that the BAM-COMP was a valid measure of comprehension by accounting for 11% of the variance in scores on the WJ-IV: ACH Reading Cluster. Additional regression analyses were conducted by regressing the WJ-IV: ACH subscales Passage Comprehension and Reading Recall onto the BAM-COMP subscales. Results showed that both BAM-COMP subscales were significant predictors of comprehension scores on the WJ-IV: ACH Passage Comprehension subscale and the Reading Cluster omnibus scale, further confirming concurrent validity of the BAM-COMP instrument. BAM-COMP was not a significant predictor of the WJ-IV: ACH Reading Recall subscale. Such results emphasize the importance of how participants may apply comprehension skills within context, and the impact such applications may have on advanced literacies and learning. Fairness of the BAM-COMP was established through four separate two-way analysis of variance (ANOVA) examining gaming habits, gender, and reading habits. Results indicated a significant difference in scores on the BAM-COMP for participants with higher reported gaming habits than those with lower reported gaming habits. Pairwise comparisons showed that a distinct threshold of gaming every day may impact scores on the stealth assessment. These results emphasize the BAM-COMP’s potential to fairly and accurately assess comprehension as an experiential behavior regardless of prior reading knowledge or abilities. Taken in aggregate, the results of the study provide empirical evidence that the BAM- COMP can be considered a valid, reliable, and fair assessment of comprehension as a holistic construct. Further, the results offer valuable insight into the nature of comprehension skills as a contextualized experience, and the impact of how applying such skills situationally differs both inter and intra-individually. As such, this study underscores the need for psychometrically sound performance-based assessment models within the field of education.