Investigation of multisource object detection fusion for improving scene awareness for enhanced autonomy

Autonomous vehicles typically employ several types of sensors simultaneously, such as LiDAR (Light Detection and Ranging) sensors, visual cameras, and radar to provide information about the surrounding scene while driving. Ideally, coupling multiple sensors would improve a system that only utilizes a single sensing modality for its perception task. For an autonomous system to understand the scene intelligently, it must be equipped with object localization and classification, which is commonly employed using a visual camera. Object detection and classification may also be applied to LiDAR and thermal sensors to further enhance scene awareness of the autonomous system. Herein, we investigate the fusing of information obtained from visual (RGB), LiDAR, and infrared (IR) sensors in order to improve object classification accuracy and heighten scene awareness. In autonomy, there are several levels of fusion that can be employed, such as sensor-level, feature-level, and decision-level; for the scope of this research, we will be exploring the impact of decisionlevel fusion. Three state-of-the-art object detection and classification models (visual-based, LiDAR-based, and thermalbased) will be employed in parallel, and object predictions will be fused using voting-based (or rules-based) fusion methods. Some questions remain as to the discrepancies that could occur: what can we do to mitigate sensor disagreement? Does the coupling of sensor decisions generally boost confidence or induce confusion? Will different fusion methods provide differing levels of impact to the final solution? Additionally, does this multi-source fusion application transfer well to different scenes? A qualitative and quantitative analysis will be presented for applications of simple and complex fusion methods, and, understanding from past research, we hypothesize that multiple modality perception algorithms boost the final solution by balancing individual sensor strengths and weaknesses. The experiments performed herein will be conducted on a novel multi-sensor autonomous driving dataset created at the Center for Advanced Vehicular Systems (CAVS) at Mississippi State University and in collaboration with the US Army’s Engineer Research and Development Center (ERDC).