Learning Cross-domain Keypoint Detection between Image and Point Cloud with Auxiliary Depth Scan Map

2D-3D cross-domain registration is the process of finding a geometry transformation aligning a 2D image and 3D point cloud to a consistent coordinate. That is, we finding the camera pose, i.e., the extrinsic of the imaging camera is the frame among the coordinate of given 3D point cloud. With the retrieve pose, we conduct various real-world applications like Augmented Reality and the localization of autonomous driving vehicles/robots. In this study, we propose geometry-aware 2D-3D cross domain keypoint detection (GCD). This is a novel approach for 2D-3D cross-domain-keypoint detection that applies a dual-branch network to predict the detection uncertainty score for each point in 2D and 3D inputs. The key to cross-domain detection is the intrinsic divergence between a 2D RGB image and a 3D geometry point cloud. Hence, we propose to exploit the dense map of 2D images in our training process to bridge the gap between the two modalities.