Amp: single-shot ultra-wide fisheye-to-cubemap PnP pose estimation

Abstract Estimating the position and orientation of a rigid object from an image is critical for situational awareness in robotics and autonomous systems. This study explores relative pose estimation using an ultra-wide fisheye camera for unmanned aircraft inspection vehicles. Ultra-wide fisheye lenses introduce radial distortion and capture features beyond the rectilinear image plane, rendering rectilinear Perspective-n-Point (PnP) algorithms inadequate. Designing a bespoke ultra-wide fisheye localization algorithm requires consideration of both the feature detection method and the pose estimator itself. This study proposes a novel method that combines (1) a fisheye-to-cubemap reprojection, (2) a You Only Look Once (YOLO) convolutional neural network trained for arbitrary airborne perspectives, and (3) an Angle-Agnostic and Multiple-Frame PnP (AMP) pose estimation algorithm. Our pipeline achieves a 97% success rate for valid pose estimates, with a mean absolute translational error of less than 12 cm on real ultra-wide fisheye imagery, outperforming conventional techniques, including OpenCV.