Are simpler math stories better? Automatic readability assessment of <scp>GAI</scp> ‐generated multimodal mathematical stories validated by engagement

Abstract Mathematical stories can enhance students' motivation and interest in learning mathematics, thereby positively impacting their academic performance. However, due to resource constraints faced by the creators, generative artificial intelligence (GAI) is employed to create mathematical stories accompanied by images. This study introduces a method for automatically assessing the quality of these multimodal stories by evaluating text‐image coherence and textual readability. Using GAI‐generated stories for grades 3 to 5 from the US math story learning platform Read Solve Create (RSC), we extracted features related to multimodal semantics and text readability. We then analysed the correlation between these features and student engagement levels, measured by average reading time per story (behavioural engagement) and average drawing tool usage per story (cognitive engagement), derived from browsing logs and interaction metrics on the platform. Our findings reveal that textual features such as conjunctive adverbs, sentence connectors, causal connectives and simplified vocabulary positively correlate with behavioural engagement. Additionally, higher semantic similarity between text and images, as well as the number of operators in the stories, is associated with increased cognitive engagement. This study advances the application of GAI in mathematics education and offers novel insights for instructional material design. Practitioner notes What is already known about this topic Mathematical stories can enhance students' motivation and interest in mathematics, leading to improved academic performance. Generative artificial intelligence (GAI) has been increasingly employed to create multimodal educational content, including mathematical stories with accompanying images, to address content creators' resource constraints. Prior readability research has primarily focused on the analysis of text‐based educational content, with less emphasis on the integration and analysis of visual elements. What this paper adds Introduces a novel automated multimodal readability assessment method that evaluates the coherence between text and images and the readability of text in GAI‐generated mathematical stories. Identifies specific story features, such as the more frequent use of three types of conjunctions (adversative conjunctions, common sentence conjunctions and logical conjunctions) and vocabulary simplicity that correlate with student engagement. Implications for practice and/or policy Educators and curriculum developers are encouraged to utilise automated multimodal readability assessment tools to analyse and refine GAI‐generated educational content, aiming to enhance student engagement and learning experience. Suggestions for the design of educational content includes the consideration of identified readability features that correlate with higher engagement. Caution should be exercised in handling the association between images and text considering the cognitive load of the instructional materials.