3D semantic segmentation for autonomous driving using late fusion of LiDAR range images and camera images with simple 2D transformation

In autonomous driving technology, surrounding environment recognition is performed using various sensors such as cameras, LiDAR (Light Detection and Ranging), and radar, and technologies such as sensor fusion, data alignment, and semantic segmentation are essential. Sensor fusion integrates multiple sensor data to achieve more accurate environment recognition. Data alignment converts different sensor data into a common coordinate system and supports sensor fusion. Semantic segmentation assigns semantic labels to each pixel in an image and identifies roads, pedestrians, vehicles, etc., allowing vehicles to deeply understand the surrounding environment. To obtain highly accurate semantic segmentation results in sensor fusion, accurate data alignment is required, which in turn requires accurate sensor calibration. In addition, alignment requires 3D coordinate transformation of each pixel in each image, which requires a non-negligible amount of computation. In this paper, we propose a simple 2D coordinate conversion alignment method for fusion of camera images and LiDAR range images, and a lightweight CNN (Convolutional Neural Network) that simultaneously performs alignment correction and 3D semantic segmentation. Experiments showed that the accuracy of semantic segmentation is improved by 7.3% for vehicles and 10.9% for pedestrians compared to the conventional method using only range images.