Category-level object pose estimation via novel feature fusion and two-stage pose regression

Abstract Category-level object pose estimation has attracted attention in robotics and augmented reality, as it enables generalization to unseen instances within known categories. However, significant intra-category shape variations and the complexity of effectively integrating geometric and visual information pose challenges to accurate pose and size estimation in vision measurement. Leveraging the widespread availability of affordable RGB-D sensors, this study explores an effective strategy for extracting and fusing appearance and geometric features to improve estimation performance. We introduce a vision foundation model for robust appearance feature extraction and propose a novel point-to-patch feature fusion strategy that leverages pixel-wise RGB-D correspondence to concatenate patch-level appearance features with point-level geometric features. The estimation process follows a two-stage paradigm, where object translation and size are first predicted, followed by rotation estimation guided by an auxiliary branch designed to facilitate learning of rotation-aware transformations. The proposed method achieves state-of-the-art performance on the CAMERA25 and REAL275 datasets, with enhanced generalization on the Wild6D dataset. Additionally, it supports real-time inference at 34 FPS on a single NVIDIA RTX A6000 GPU.