LiDAR and Camera Fusion for Joint Depth Completion and Panoptic Segmentation Tasks in a Unified Network for 3D Semantic Segmentation

Autonomous driving relies heavily on advanced perception systems built on multi-sensor fusion, including LiDAR, cameras, GPS, and IMU. Among these, LiDAR is particularly valuable for providing accurate 3D spatial information, making it essential for many autonomous driving algorithms. However, LiDAR also presents challenges such as data sparsity, limited detection range (especially beyond 70 meters), and high computational costs. These limitations hinder the system's ability to detect distant objects or respond to sudden appearances and abrupt stops in real-time scenarios. In contrast, cameras offer rich texture and color information but struggle with depth estimation. To address the shortcomings of each sensor while leveraging their respective strengths, this paper proposes an integrated network architecture that combines LiDAR and camera data through early fusion. We introduce a lightweight, efficient fusion network capable of performing high-precision camera-based Panoptic Segmentation and depth completion simultaneously, thereby enhancing sparse LiDAR data. The proposed architecture achieves over 9 FPS inference speed on a single RTX 4070 GPU. For real-time 3D Semantic Segmentation, we fuse outputs from each task and deploy the system using ROS2 and Autoware. Experimental results demonstrate that our model effectively balances performance and computational efficiency. Its ability to handle a variety of perception tasks with fewer resources makes it suitable for practical deployment in autonomous driving and 3D scene reconstruction applications.