Fusion-Based Utilization and Synthesis of Efficient Detections (Project F.U.S.E.D.)

Sensor systems are essential for a variety of aerospace applications. Drones, airplanes, and satellites are outfitted with arrays of sensors that collect data, fulfilling missions and enabling a better understanding of the operating environment. Some examples of sensors used for such applications include cameras, radars, and LiDAR (Light Detection and Ranging) sensors. With recent AI developments, the use of object detection in camera systems has become prevalent, allowing for useful information to be derived from camera images without any real human input. Object detection is a subset of machine learning in which models are trained to label and put boxes around objects of interest in images. Such a system proves useful when deployed on cameras. The following scenario was considered: If an autonomous drone was deployed in a crisis response situation to search for people, would a single camera provide enough information and confidence if the AI systems indicated that a person was spotted? The answer, in many situations, is no. There would be several scenarios in which a simple webcam (color camera) would be useless. What if it was dark outside? What if it was very foggy? What if the drone was in an urban environment with people projected on screens or mannequins standing in windows? It becomes apparent that a combination of sensors with different strengths is required for this task to be accomplished. Combining sensor data together, termed sensor fusion, is an active area of research in many disciplines in which sensor data is merged in useful ways. The research in this paper aimed to develop a sensor system, both by assembling hardware and software, that would deploy object detection models in a sensor fusion framework. Three sensors were used: a LiDAR sensor, a thermal camera, and a webcam. Although much has been done by researchers investigating methods for sensor fusion between cameras and even LiDAR systems, the aim was to experiment with a sensor fusion method specifically intended for efficient, lightweight operations. The goal was to build the system and prove with quantitative results that the method of fusion was more robust than individual sensors using object detection models. The system hardware was purchased, assembled, and calibrated, and the system software was developed. In brief, the software method aimed to take LiDAR depth image detections, retrieve depth data from the centers of the box detections, perform coordinate transformations to map the LiDAR detections onto the webcam and thermal camera coordinate frames, compare the boxes, and then reach a decision based on which sensor detections overlap. Other methods may aim to transform all pixels from one image into another image, or perhaps stereo imaging techniques may be employed to resolve depth information. The proposed method was chosen due to its simplicity and its seeming ability to quickly and efficiently provide reliable detections of people. The intention was for it to be simple to implement and easily deployable on smaller systems, such as embedded computers. Alongside the hardware and software development, sensor data was manually gathered and labeled for model training. The experiment aimed to detect people in a variety of challenging situations, as the scenario detailed earlier. Some of the different situations included recording a dummy model of a person, recording in complete darkness, and recording people projected on screens. Control data was also collected in ideal conditions, and the goal was to only detect warm bodies. Once the models were trained, test video recordings from the gathered data were used on the fusion algorithm for viewing detections. The output from the algorithm was compared with detections from models deployed on the individual sensors, outside the fusion algorithm loop. When viewing the results, it was clear that the fusion algorithm performed more reliably than the individual sensors. Quantitative analysis is still in process, but it is already clear that the fusion algorithm performed as expected, proving that the fusion method proposed has value for certain situations. Future work may consider real-time deployment of the system on an embedded computer, more accurate methods of calibration, more reliable methods of quickly recovering depth from the LiDAR images, or possibly deploying localization tools on the sensor system for recovering precise location data. Such location information could be used in the algorithm to determine locations of the detected objects.