ZOO: A Training-Free Framework for Open-Vocabulary Animal Detection, Segmentation, and Tracking

Abstract Quantitative extraction of animal visual information often relies on annotated data collected for specific species and recording environments, limiting rapid transfer across biological systems. We present ZOO (Zero-shot Organism Observation), a zero-shot animal visual analysis workflow that requires no additional training or fine-tuning on target species or recording environments. ZOO integrates the open-vocabulary detector Grounding DINO with the segmentation and video propagation model SAM-2 to extract bounding boxes, instance masks, and spatiotemporal trajectories from images and videos. It includes ZOO-Image for static images and ZOO-Video for video sequences, aiming to provide standardized visual information for downstream behavioral analysis with a low deployment barrier. We evaluated ZOO on hierarchically labeled datasets, high-density animal scenes, and animal videos. ZOO-Image performed comparably to Grounding DINO in images containing few animal instances, but effectively reduced false positives and improved precision in some high-density or less-occluded scenes. Additional experiments examined the effects of hierarchy-derived text prompts on zero-shot detection performance. ZOO-Video achieved competitive multi-object tracking performance without target-domain retraining while simultaneously producing trajectory-associated instance masks. Overall, ZOO provides a convenient framework for cross-species detection, segmentation, and tracking without target-domain annotations.