Multimodal Writing Evaluation in Digital Storytelling using Video-Based Output: Comparing performance of AI and Human Raters.

With the rapid advancement of artificial intelligence (AI) models, it is now possible to perform content analysis on multimodal videos using this kind of technology. This study investigated the consistency between AI assessments and human evaluations of multimodal writing tasks, specifically comparing the scores assigned by AI models with those assigned by human raters. The research sample is composed of five multimodal writing assignments from a second language learning classroom. We compared the scores from AI and human raters using intraclass correlation coefficients (ICC) and within-group agreement reliability (rwg). The results showed minimal differences between the scores from the AI and human raters. Mean rwg values revealed a high level of agreement between human and AI assessments, whereas ICC values indicated slightly lower consistency compared to human raters. Additionally, human raters provided polarized responses to the feedback generated by AI. These findings provide evidence that AI could serve as a supplementary tool for assessing multimodal writing tasks.