A robust 3D position estimation and tracking method of multiple fish for occlusion and mirror image in a small tank

Animal populations exhibit adaptive changes in response to stimuli. A time series of individual positions and poses must be tracked to reveal collective animal behavior. Tracking fish in 3D space is needed, especially for fish. However, tracking in camera images involves the difficulties of occlusions and mirror images. This study uses three cameras to propose a robust 3D position estimation and tracking method of multiple fish for occlusion and mirror image in a small tank. Fish occlusion occurring in one direction is handled by estimating the 3D position using results from the other two directions. DeepLabCut for multiple animals (maDLC) finds fish feature points for each camera view. Post-processing performs for proper correspondence between feature points and individuals: Removing misassignments, matching the feature point in all camera views, and merging feature points representing the same fish. Feature points are estimated in 3D, and if their centroid is outside of the tank, they are eliminated as mirror images. Experimental results showed that even if fish occlusion occurred in one direction, 3D position estimation could be performed if feature points were estimated in the other two directions using the proposed method. Mirror images were eliminated, and the actual fish could be tracked.