MFICNet: A Multimodality Fusion Network With Information Compensation for Accurate Indoor Visual Localization

As a crucial technology in numerous visual applications, visual localization has been extensively studied, with an effective solution known as scene coordinate regression (SCoRe). Generally, SCoRe methods generate scene coordinates using convolutional neural networks (CNNs) and then determine the camera pose with a PnP algorithm. While these methods demonstrate impressive localization accuracy, they primarily rely on a single modality, e.g., RGB camera, which leads to texture dependency and structural ambiguity problems. Specifically, perceptual confusion caused by similar image textures in real indoor scenes causes a severe decline in localization accuracy, as the performance of the networks heavily depends on the semantic information of objects. In addition, current methods struggle to robustly recover the structural details of objects because RGB images lack 3-D geometric structural information. We think that these two issues stem from the inherent limitations of single modality. There is potential for complementarity between semantic and structural information. Toward this end, we propose MFICNet, a novel visual localization network that simultaneously utilizes RGB and depth images to achieve accurate visual localization. This pioneering architecture establishes a new paradigm for multimodality-based visual localization. Technically, MFICNet employs a heterogeneous backbone to extract features from RGB images and depth images separately. The structural feature obtained from depth images enhances the identifiability of similar image patches and imposes structural constraints for scene coordinates. After that, an information compensation module is introduced to evaluate the contributions of semantic and structural features and perform deep fusion to generate discriminative features. Extensive experiments are conducted on the 7-Scenes dataset and our newly released indoor dataset STIVL, which specializes in similar textures. The results show that MFICNet significantly outperforms state-of-the-art (SOTA) methods. The source code and STIVL dataset are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/fazhdo/MFICNet</uri>.