Dog's 3D Skeleton Reconstruction using a Moving Trainer for Analysis of Guide Dog Training

This study aims to enhance the training efficiency of guide dogs by employing computer vision to collect training data and analyze the movements of both trainers and dogs. This task is challenging, owing to the constant movement of cameras and unstable reference points for camera calibration, which stem from the complexities of the guide dog training process and the surrounding environment. In addition, trainers and dogs walk side by side, making it difficult for 2D videos to capture complete interactions without obstacles. We present a comprehensive system that starts with 2D video footage from multicamera setups, proceeds to extrinsic camera calibration from the moving trainer's joints, and reconstructs the 3D poses of guide dogs and trainers. This process includes human 2D/3D pose estimation, camera calibration, and dog 2D/3D pose estimation. A novel aspect of the proposed approach involves modifying the existing calibration method for multiple cameras. This modification is designed to achieve extrinsic camera calibration and accommodating complex camera settings in real-world situations, including both fixed and moving multicamera setups without calibration objects. We can create a 3D representation of the training sessions by detecting the trainer's 2D and 3D skeletons and using calibrated cameras to triangulate the dog's 3D pose. This allows for a detailed analysis and adjustment of guide dog training methods based on the 3D pose data of trainers and guide dogs, thereby improving the overall training process.