Robust Place Recognition Using Fusion of Thermal Infrared and RGB Cameras

The performance of camera-based place recognition has significantly improved with the rapid advancement of deep learning. However, RGB cameras still face challenges in handling variations in lighting conditions due to their inherent limitations. Recent research has explored the use of thermal infrared (TIR) cameras to achieve illumination-invariant scene recognition. Nevertheless, TIR cameras can also struggle in scenarios with uniform heat distribution or when objects with extreme temperatures are present. We herein propose a robust place recognition methodology that leverages RGB-TIR feature fusion. We developed a novel network architecture that can dynamically extract information weights from each RGB and TIR image and then generate a weighted descriptor. This approach is particularly effective in scenarios in which one of the sensor modalities is degraded. The proposed method was validated using a dataset covering a subterranean environment, and it demonstrated superior performance compared with existing visual place recognition (VPR) methods. Additionally, we conducted an ablation study to evaluate the effectiveness of the proposed weight extraction network, providing qualitative insights into its performance.