Temporal aggregation of vision-language features for high-accuracy fish classification in automated monitoring

This work demonstrates how contrastive language-image pre-training models, using the SigLIP architecture and prompt engineering optimization, can effectively classify fish species without frame-level annotations or preprocessing in controlled underwater environments. The study developed a comprehensive pipeline to detect and classify three salmonid species (Atlantic Salmon, Trout, and Char) from Icelandic river monitoring footage, achieving impressing results, exceeding the performance of a fine-tuned ResNet50 model: 99.1% accuracy in fish detection, 98.2% accuracy in multiple fish detection, and 97.5% accuracy in species classification using SVM with temporal pooling. The system employs zero-shot learning combined with temporal feature aggregation strategies, requiring significantly less training data than traditional approaches while maintaining robust performance. The primary contribution demonstrates that embedding models can play an important role in automated marine and freshwater biology monitoring systems. • SigLIP models classify fish species with 96.8% F1-score, no annotations needed. • Pipeline detects single (99.1%) and multiple fish (98.2%) via zero-shot prompts. • Temporal pooling boosts accuracy vs. single-frame while staying efficient. • Distinguishes salmon, trout, and char with less data than traditional methods. • Handles lighting, turbidity, and pose changes with robust temporal features.