ActRecognition-GPT: Utilizing Multimodal Large Language Models for Spatiotemporal Action Recognition in Nursery Videos

Spatiotemporal action recognition in nursery videos is essential for intelligent childcare systems that can automatically monitor children’s behaviors, detect safety risks, and generate accurate activity logs without increasing caregivers’ burden. However, occlusions, dynamic multi-person interactions, and subtle distinctions between similar actions present significant challenges. To address these challenges, we introduce ActRecognition-GPT, a multimodal framework that combines visual object detection with the reasoning capability of multimodal large language models (MLLMs). By assigning unique person IDs and incorporating both spatial and temporal context through structured prompts, the model achieves temporally consistent recognition of individual behaviors, even under visual ambiguity or occlusion. Moreover, predefined vocabularies are used to constrain generation and reduce semantic drift. Crucially, the open-vocabulary nature of MLLMs enables ActRecognition-GPT to generalize to previously unseen or ambiguous scenarios without requiring extensive labeled data, which is an essential property for modeling the diverse and evolving behaviors of children. Experimental results on a real-world nursery video dataset demonstrate a significant improvement in mean Average Precision over single-frame baselines, validating the effectiveness of combining structured visual inputs with LLM-based temporal reasoning.