VIFPE: Multimodal Fusion of Visible and Infrared Images for Pose Estimation in Large-Scale Urban Environments

Most existing deep learning-based camera pose estimation methods rely solely on visible images, making them vulnerable to challenges such as drastic lighting variations and poor nighttime imaging quality. These limitations reduce the robustness and accuracy of traditional approaches. To address these issues, we propose VIFPE (Visible- and Infrared-Fusion-Based Pose Estimation), a novel framework that integrates features from both visible and infrared images to enhance pose estimation in urban environments. By leveraging the complementary advantages of visible images, which are rich in texture, and infrared images, which are resilient to illumination changes, VIFPE achieves robust and accurate localization across diverse lighting conditions. The framework consists of multiple key modules, including a feature extractor, a multimodal fusion module, a pose regressor, and a multi-task learning strategy. Specifically, VIFPE employs Multi-axis Vision Transformer (MaxViT) encoders to extract features from both modalities, which are then fused in a shared feature space through a cross-modal fusion module. A pose regression network is subsequently trained to estimate the camera’s position and orientation based on the fused representation. Our experimental results on diverse urban datasets demonstrate that VIFPE significantly outperforms conventional methods in terms of both accuracy and robustness, particularly under challenging lighting conditions. This work underscores the potential of multimodal fusion for camera localization in large-scale urban environments and lays a foundation for future advancements in the field.