Thermal Image-to-LiDAR Depth Transformation via Pretrained Visual Model and Two-Stage Depth Refinement

LiDAR sensors provide reliable physical distance measurements using laser signals, enabling accurate acquisition of 3D information for various optical systems. However, they are costly, require significant weight and space, and their reliability and accuracy degrade under adverse environmental and weather conditions. In contrast, thermal cameras operating in the infrared spectrum can capture stable visual information even in challenging scenarios such as nighttime, low-light, and rain. However, they cannot directly provide the physical 3D depth information that LiDAR offers. To design efficient optical systems, there is a growing need for techniques that transform thermal image data into LiDAR-like depth information. While deep learning models can theoretically learn direct mappings between thermal and LiDAR modalities, the scarcity of acquiring paired thermal–LiDAR datasets and the difficulty of acquiring them make this task challenging. In this paper, we propose a thermal image-to-LiDAR depth transformation framework. Our method leverages large-scale pretrained visual models for depth estimation to generate initial depth predictions from thermal inputs. Since pretrained RGB-based models face a modality gap when applied to thermal data, we introduce a two-stage depth refinement. Stage 1 corrects global scale inconsistencies, and Stage 2 refines local structural details. Experiments on the MS2 dataset demonstrate that the proposed framework consistently improves the initial DepthPro outputs across day, night, and rainy conditions. Both quantitative metrics and qualitative comparisons show that RGB-pretrained depth predictions can provide useful structural cues for thermal depth estimation when their global scale and local structural errors are explicitly refined.