CaRaCTO-3D: From Camera–Radar Calibration to Scene Reconstruction

The multimodal nature of camera and radar sensor data enables various automation and surveillance tasks, where one sensor compensates for the limitations of the other sensor: cameras capture high-resolution color data, while radar measures depth and velocity of targets. Calibration is essential to fuse these data modalities effectively. This work presents a robust extrinsic calibration algorithm for camera–radar setups, extending standard geometric constraints with elevation information to enhance optimization. Unlike existing methods, this approach relies solely on camera and radar data without requiring complex targets or external measurements. The 3D calibration enables the estimation of the target elevation which is lost when using 2D radar. We evaluate our results against a sub-millimeter ground truth system, demonstrating superior performance compared to more complex algorithms. Leveraging these accurate calibration results, we subsequently employ monocular depth estimation and instance segmentation techniques to perform camera–radar data fusion, allowing 3D target and scene reconstruction. github.com/mahdichamseddine/CaRaCTO .