Generalized differentiable Perspective-n-Point without 2D-3D correspondences for pose measurement

Abstract Solving the Perspective-n-Points (PnP) problem without two-dimensional (2D)-three-dimensional (3D) correspondences is known as blind PnP. The extensive search space and numerous outliers present significant challenges in accurately measuring the camera pose. We proposed a generalized differentiable blind PnP method for real-time, high-precision camera pose estimation and measurement. The method introduces an innovative graph neural network for cross-modal matching of 2D and 3D points, jointly identifying correspondences and rejecting outliers. Assignments are estimated by solving a differentiable optimal transport problem, with dustbin channels added to handle outliers effectively. The probabilistic PnP layer facilitates end-to-end learning and inference of pose, leveraging 2D-3D joint correspondence probabilities. Furthermore, geometric constraints and weighted nonlinear PnP solvers are employed to optimize the assignment matrix globally. Experimental results show that our method surpasses existing algorithms in accuracy and speed for blind PnP problems, particularly with large-scale datasets, highlighting its potential as a generalized PnP solver. In comparison to laser trackers for robot arm positioning, the proposed method exhibits an error level of 10 −1 mm, suggesting its viability as a cost-effective and efficient substitute.