EPAF-DETR:Efficient transformer-based model for abnormal fish behavior detection under water quality anomalies

• Improved RT-DETR model (EPAF-DETR) for detecting abnormal behavior in grouper in aquaculture. • The model demonstrates good detection performance and robustness in complex farming environments. • Design new feature extraction modules and attention fusion modules to improve model accuracy. • Surpassed the original model and mainstream YOLO algorithms in terms of F1 score and average accuracy. • Providing real-time, efficient, and accurate solutions for monitoring fish health in aquaculture. In aquaculture management, the prompt recognition of abnormal fish behaviors induced by external stimuli or diseases is crucial for enhancing breeding efficiency and securing economic returns for farmers. Nevertheless, monitoring such behaviors remains challenging owing to intricate aquatic environments, frequent occlusions, and considerable visual similarities across different behavioral categories. To tackle these issues, this study introduces an Efficient Parallel Attention Fusion Detection Transformer, designated as EPAF-DETR, which is developed to achieve high-precision and robust object detection. By integrating EfficientViT as the backbone network, the computational complexity of the model is significantly reduced. Combined with an adaptive sparse self-attention mechanism and a spatially enhanced feedforward network, an improved AIFI module is introduced to strengthen feature extraction capabilities. Furthermore, Multi-Level Hierarchical Attention Fusion module is designed to enhance the original cross-scale feature fusion component in RT-DETR, enhancing the salience of critical features and further improving detection accuracy. Finally, by incorporating Matchability-Aware Loss function, the model is guided to place greater emphasis on matching low-quality features.These architectural advancements considerably boost the model’s adaptability in demanding underwater settings and augment its capacity to discriminate fine-grained behavioral characteristics of fish. Experimental outcomes indicate that EPAF-DETR attains detection performance while reducing computational costs, achieving an average F1-score of 94 % and a mAP of 95.7 %. In conclusion, the proposed approach effectively addresses detection difficulties in complex aquaculture environments, enabling accurate and reliable identification of anomalous fish behaviors.