Geometrically constrained permutation invariant learning for hand-eye calibration with iteratively reweighted least squares and Levenberg–Marquardt refinement

Purpose This paper aims to develop an accurate and robust hand-eye calibration framework for estimating the rigid transformation between a camera and a robot end-effector, particularly under non-ideal conditions involving noisy pose measurements, outliers and motion pairs of varying quality. Design/methodology/approach The proposed method models unordered relative pose pairs using a shared, attention based, permutation invariant encoder. Rotations are represented in the Lie algebra and mapped to SO(3) via the exponential map, while translations are estimated by least squares. The resulting initial estimate is then refined on SO(3) using iteratively reweighted least squares combined with Levenberg–Marquardt optimization, thereby improving robustness to measurement noise and outliers. Findings Simulation and experimental results demonstrate that the proposed method achieves accurate and robust hand-eye calibration. Geometry aware refinement enhances calibration stability in the presence of noise and outliers while preserving the physical consistency of the estimated rigid transformation. Originality/value This study develops a permutation invariant hand-eye calibration approach that integrates Lie group-based rotation modeling with robust refinement on SO(3), enabling reliable calibration under noisy and heterogeneous measurement conditions.