Reflection-aware 3D mirror segmentation and pose estimation

Abstract Reflective surfaces are notoriously difficult to detect and reconstruct, as they reflect light from surrounding objects. Thus, the quality of 3D reconstructions and the performance of downstream computer vision tasks, such as navigation, depth prediction, or object detection are severely impaired by the prevalence of mirrors in indoor environments. This paper proposes a novel reflection-aware method for 3D mirror segmentation and pose estimation, based on the reinterpretation of captured RGB-D data. It is a top-down approach, where information consensus, unsupervised learning, and ray casting are employed, formulating mirror pose estimation as an optimization problem. Experimental results show that the proposed approach significantly outperforms the existing benchmarks for RGB and RGB-D-based 3D mirror segmentation and pose estimation.