Trinocular vision with deep learning for object twist estimation: a benchmark approach and dataset

Accurate estimation of a rigid object’s 6D pose and twist is a fundamental challenge for enabling autonomous systems operation and interaction with the environment. Monocular vision can mitigate the drift inherent in inertial methods and enables the estimation of non-cooperative target twist. However, the estimation accuracy of monocular vision is compromised by inherent depth ambiguity, a limitation that multi-camera systems can overcome. The lack of multi-view datasets with high-precision annotations hinders the development of robust perception algorithms. To address this, we present the first trinocular pose and twist estimation dataset for non-cooperative targets, comprising images of aircraft (7,824 for training, 4,710 for testing) and satellites (7683 for training, 4380 for testing), all manually annotated with sub-pixel-level keypoints and pose labels derived from optimization. We develop a neural network that predicts semantic keypoints for robust pose estimation. Combined with a multi-view optimization framework and twist estimation, our system achieves a mean angular velocity error of $$0.1^{\circ }$$/s and a mean linear velocity error of 0.3mm/s. Our open-source dataset and method provide a critical benchmark for future research in aerospace missions. We have open-sourced the dataset at the following https://www.kaggle.com/datasets/mingshiwuwjh/trinocular-pose-and-twist-estimation-dataset.