DTDepth: A Self-Supervised Framework With Ultrasparse LiDAR for Image Depth Estimation

Self-supervised learning provides a promising approach for monocular depth estimation (MDE) tasks by mitigating the reliance on training data that is both scarce and expensive to acquire. However, prevailing image-based and multi-modal self-supervised MDE methods often fail to attain the high accuracy demanded by downstream applications. This study explores a self-supervised method leveraging the guidance from ultra-sparse LiDAR, to enable an effective and cost-efficient framework for dense depth prediction. Inspired by the convolutional neural network (CNN) and Vision Transformer (ViT) models, we propose DTDepth, a novel hybrid framework based on dilated convolution and transformer operators. Given the image and low-beam LiDAR as input, we designed an efficient fusion module in our encoder for the deep interaction between the dense image and sparse LiDAR features, then we extensively utilized attention mechanisms to enrich the aggregated features in the decoder. The key part of our framework is Joint Transformer and Dilated Convolution (JTDC) block, which can effectively consider global and local feature extraction. Departing from traditional dense depth completion, our innovative approach employs affordable low-beam LiDAR to guide the image depth prediction. We evaluated the performance of our model on KITTI and Make3D datasets, where we demonstrated that our proposed method significantly outperforms many existing state-of-the-art algorithms and narrows the performance gap between supervised and self-supervised methods.