Investigating Children’s Multimodal Enactment in Digitally Augmented Tabletop Storytelling

Young children often struggle to effectively express their ideas in both speaking and writing due to their developing cognitive and linguistic abilities. To address this challenge, this paper introduces the Digitally Augmented Tabletop Enactment (DATE) approach that integrates tactile-kinesthetic, augmented visual, and auditory elements to support motivation and multimodal enactment in storytelling by enriching children’s interactive experiences and fostering dynamic information processing across multiple sensory modalities. To implement this approach, we developed the lightweight Tabletop Imaginative Play as Enactive Storytelling (TIPES) system that combines physical play on a tabletop with real-time digital story representation. This system captures children’s manipulation of figurines and translates their actions into animated scenes, empowering them with direct control over story content and instant visual feedback. We conducted a user study with 36 children to evaluate our approach’s impact compared to traditional tabletop storytelling. Our analyses examined the children’s embodied interactions during story enactment, writing performance, and user feedback. Results revealed that the DATE approach supported a stronger integration of physical and vocal modalities, greater narrative agency, and strengthened eye–hand coordination, reflecting deeper engagement. Importantly, children’s performance in verbal tasks such as speech narration and writing remained comparable across conditions, indicating that multimodal enactment along with an additional digital authoring experience can enrich storytelling without compromising narrative output quality. Additionally, children expressed strong enthusiasm for the DATE approach, underscoring its appeal in sustained engagement in creative storytelling. These findings highlight the potential of physical–digital tabletop enactment as a promising direction for designing educational technologies that scaffold expressive, collaborative, and developmentally appropriate narrative practices. • Introduced the DATE approach to scaffold children’s storytelling. • Analyzed multimodal interactions across physical actions, speech, and eye gazes. • Found enhanced engagement with balanced modalities and stronger agency. • Showed added authoring tasks did not hinder narrative performance. • Promoted appreciation on the DATE approach supporting sustained literacy development.