Real-Time Rat Pose Estimation System via Miniature Stereo Vision for Robot-Rat Interaction

Robot–animal interaction has been gaining attention in neuroscience and robotics, which leverages biomimetic and robotic techniques to gain insight into rat individual and social behaviors. However, existing interactive robots still struggle to make appropriate decisions by perceiving the behavior of interacting objects in real-time. Here, we proposed a real-time rat pose estimation system for robot autonomous interaction and behavioral analysis. A miniature binocular camera embedded in the head enables the rat-like robot SMuRo to adjust its viewpoint and track moving rats based on visual information. To ensure accurate and fast rat keypoints detection for multirat scenarios across various robot–rat distances, we developed a real-time rat keypoint detection network (RRKDNet) with a multiresolution fusion structure, which performs with optimal accuracy, speed, and cross-scale stability in our experimental scenarios. The framework was further combined with a rat-ID fast-matching method and an optimization-based 3-D keypoints reconstruction process, realizing simultaneous 3-D pose estimation for multiple rats. We demonstrated that the proposed system could accurately estimate the 3-D poses of multiple rats in real time. In addition, robot–rat interaction experiments show that real-time pose information helps SMuRo to switch behaviors with appropriate timing. The percentage of actual interactions during the ideal interaction period reached 62.5%. This research paves the way for robot–rat group interaction and can easily be extended to biomimetic interactive robots inspired by other animals.