42 Automated segmentation and tracking of group housed pigs using zero-shot vision-language tools.

Abstract Accurate individual tracking in group-housed pigs is critical for high-precision phenotyping, enabling novel trait development. Current approaches for automatic pig tracking rely on video recordings analyzed by computer vision models, which require labor-intensive manual video annotation and costly model optimization. Recent advancements in zero-shot learning, especially through the integration of large vision models and language models, enable automated annotation and precise context-aware tracking. Grounded-SAM2 is a state-of-the-art vision-language model that ensembles Grounding-DINO and Segment Anything Model 2 (SAM2) to perform detection, segmentation, and tracking based on textual prompts. The objective of this study was to assess the merits of zero-shot artificial intelligence technology for the automatic tracking of individual pigs. Six top-view cameras were installed overlooking six pens of nursery pigs to record videos from 06:00 to 17:00 each of two days. The pigs used in this study were crossbred white pigs (L42 x L337, PIC), with an initial body weight of 5.86 ± 1.01 kg. Pigs were housed at a density of ten animals per pen and they were marked with painted numbers on their backs. From every 64 minutes, we extracted the first 5-minute-long video that was divided into 1-minute segments for tracking, resulting in a total of 550 segments. From each video, we extracted the first frame and used Grounded-SAM2 with the prompt “pig” to detect pig bodies with bounding boxes and segmentations, propagating them across subsequent frames to generate tracks. The tracks of individual pigs were recorded as auto-annotated videos (4,927 videos comprising 2,956,200 frames), with each video containing a single tracked pig, highlighted by its assigned mask. A human evaluator watched every 1-minute auto-annotated video and recorded the presence of one or more of four errors: incorrect mask (incorrect segmentation), duplicated label (same pig detected multiple times), ID switch (track switch between pigs), and lost track (failure to maintain a track). Completely correct tracks were observed in 87.38% ± 0.91% of videos on the first day and 83.74% ± 1.20% on the second day. Error rates on the first day were 6.65% ± 0.69% (incorrect mask), 6.80% ± 0.69% (duplicated label), 2.27% ± 0.41% (ID switch), and 0.45% ± 0.18% (lost track). On the second day, errors increased slightly: 8.93% ± 0.93%, 8.40% ± 0.90%, 2.02% ± 0.46%, and 0.21% ± 0.15%, respectively. These results are remarkably impressive, considering that no training or fine-tuning was performed, and the test videos are an entirely unseen dataset compared to the training data used for Grounded-SAM2.In conclusion, the zero-shot learning vision-language model proved to be a valuable tool for automatically segmenting and tracking pigs in group settings during the daytime without the need for manual annotation and task-specific fine-tuning.