3D behavior phenotyping and multi-object tracking system of mice based on deep learning

• A 3D behavioral analysis system suitable for observation of unmarked mice (also without RFID) in natural living condition. • A 3D behavioral analysis pipeline including a 3D point clouds data preprocessing method, a 3D point clouds segmentation method, and an improved multi-object tracking method for 3D point clouds. • The proposed pipeline was used to observe the behavior change of muscle atrophy mice, and we found that muscle atrophy mice had significant differences in movement ability, diet and body posture compared with normal mice. • A low-cost system was developed which was suitable for short/long observing of one/ multiple small animals and large scale experiments. To extract multiple-mice behaviors automatically, nondestructively, and long-termly, a 3D behavior extraction system was developed for extracting individual behaviors and social behaviors of mice. In this study, two depth cameras were used to acquire 3D information of mice. 3D point clouds model of mice was established and the algorithms of clipping, statistical filter, matching, plane segmenting, and rotating were used to preprocess the 3D point clouds. In this study, YOLOv7 was used to detect mice and 3D Kalman Filter, Hungarian algorithm, and improved matching algorithm were used to track mouse identity. Stratified Transformer was used to segment individual mouse from the environment and segment individual mouse into 3 parts, and 8 feature points were extracted. In individual mouse experiment, the whole individual mouse was segmented to 4 parts and 11 feature points were extracted. Individual and social behaviors of mice were extracted combined feature points and mice trajectories. The results showed the accuracy and average mIoU of individual mouse segmentation were more than 95.80% and 91.90%, respectively, and the accuracy and average mIoU of parts segmentation were 93.60% and 85.9%, respectively. Compared with manual measurement of mice weight, the determination coefficient (R 2 ) of mouse volume was 0.9726. The maximum absolute error of the automatic measurements versus manual measurements of mouse tail length was 0.4 mm. The precision, recall, and mAP of YOLOv7 were 99.76%, 99.53%, and 99.75%, respectively. The analysis of behaviors also showed that there were significant differences between dexamethasone-induced muscle atrophy and normal mice.