PixelFusion: A Pixel-level Multi-Sensor Fusion Method for 3D Object Detection

Multi-sensor fusion stands as a pivotal approach for enhancing the efficacy of 3D object detection in the context of autonomous driving. Nowadays, the fusion methods are mainly based on BEV (Bird's-Eye View). This kind of methods benefits from the advantage of BEV features fitting geometry size and distance, but they are limited by BEV's weakening of camera semantics features and limited detection range. In view of this situation, we propose a 3D object detection framework based on visual stream, dubbed PixelFusion, which unifies the point clouds and semantic images into the pixel-level unit of the image. It avoids the limitation of BEV-based methods, which often discard the semantics density of camera features and are constrained by the BEV range. In order to achieve a better fusion result, we also introduce a Dynamic Feature Selection module designed to optimize the integration of image and point cloud data. Our experimental results show that, after training, our framework outperforms several classical 3D object detection methods. To the best of our knowledge, this is the first method to convert Lidar point clouds into 3-channel pixel-level images.