Experimental Analysis of Animal Behavior for Biomedical Applications

This study addresses the problem of robust video-based tracking of laboratory rats in open-field and Y-maze experiments under challenging acquisition conditions, including non-uniform illumination, low contrast, and heterogeneous recording setups. Existing approaches based on classical image processing or deep learning often fail to maintain stable localization under such conditions or require large, annotated datasets. We propose a hybrid tracking framework that combines an improved motion–appearance voting mechanism with consistency-constrained optimization for open-field experiments, together with a comparative deep learning-based detection strategy for Y-maze analysis. The proposed method introduces (i) adaptive dual-threshold motion extraction, (ii) directionally constrained temporal validation, and (iii) a robustness-driven fusion of motion and appearance cues. Experimental results demonstrate that the proposed approach achieves reliable tracking with a maximum localization error below 10 pixels under severe illumination variations. In the Y-maze scenario, a comparative evaluation of multiple detectors (YOLOv5, YOLOv9, YOLO12, Faster R-CNN) highlights the trade-off between accuracy and inference time, with YOLOv9 providing the best balance. The main contribution consists of enabling robust behavioral quantification in low-quality experimental conditions using limited training data, bridging the gap between classical tracking robustness and deep learning flexibility.