Spatiotemporal video encoder integrated with metadata fusion for action recognition and behavior analysis

Automated behavior monitoring and action recognition is challenging in dense, dynamic environments of broiler pens, combined with the challenges of handheld smartphone footage. This study developed and validated an integrated pipeline for spatiotemporal action recognition and behavioral analysis of individual broiler chickens using smartphone-captured video. Our end-to-end framework addresses key technical challenges by first stabilizing raw footage to correct motion artifacts, then employing YOLOv11-Pose for robust detection and pose estimation in crowded scenes, and ByteTrack for multi-object tracking. The core innovation is a novel metadata fusion strategy, where spatiotemporal features from an X3D video encoder were integrated with pose-derived contextual metadata (such as velocity and proximity to feeders and drinkers) using a multilayer perceptron. This intermediate fusion approach enabled the model to disambiguate visually similar but contextually distinct actions, such as drinking versus feeding. The system demonstrated strong performance on five key maintenance behaviors (resting, walking, standing, drinking, and eating), achieving an average F1-score of 0.864 and an overall test accuracy of 86.41 %. The system was particularly effective at detecting drinking and feeding behaviors, with individual F1-scores of 0.949 (94 % accuracy) and 0.928 (91.89 % accuracy) respectively. The system successfully processed 174 videos, producing valuable insights in the form of behavioral metrics, spatial heatmaps, and individual trajectories. This demonstrates that an integrated system, combining stabilization, advanced multi-task models, and contextual metadata fusion, can effectively use smartphone footage to create a scalable, objective tool for precision poultry farming and welfare research. The key advantages of the proposed framework include its ability to disambiguate visually similar birds and behavior actions through metadata fusion, its use of accessible cost-effective smartphone data, and its balance of accuracy and computational efficiency.