VIL-PPGen: A Novel Pseudo Point Generator Based on Visible Light Camera, Infrared Camera and Lidar

Recently, pseudo-point-based 3D object detection framework has demonstrated superior performance and attracted significant attention. Unfortunately, current pseudo point generators, the core of this framework, fail to consider accuracy, robustness, and efficiency simultaneously. To meet these challenges, we propose a novel pseudo point generator named VIL-PPGen, which leverages a visible light camera, an infrared camera and a lidar to specifically cater to the requirements of pseudo-point-based detectors. The proposed VIL-PPGen mainly consists of three modules: Dual-spectrum Depth Completion Module (DDCM), Sparse Cost Volume Module (SCVM) and Adaptive Depth Correction Module (ADCM). The DDCM adopts a dual structure for depth completion on different spectrums to maintain accuracy, the SCVM utilizes sparse operations for cost volume computation to improve efficiency, and the ADCM employs adaptive confidence and offset for correction to reinforce accuracy and robustness. Ultimately, we can acquire high-quality pseudo points under all-day lighting conditions, which can directly improve the performance of subsequent detectors. To validate the efficacy of our designs, we construct a dataset from real driving scenarios and conduct extensive experiments. The proposed VIL-PPGen achieves 1.083m MAE (improved by 0.581m) for depth prediction and 53.10% mAP (improved by 3.75%) for multi-class detection in day scenes, while achieving 1.025m MAE (improved by 0.551m) for depth prediction and 55.64% mAP (improved by 3.94%) for multi-class detection in night scenes. It also exhibits excellent real-time performance, achieving a processing speed of 10 FPS. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Aiuan/VIL-PPGen</uri>.