Decision-Level Fusion of Electro-Optic and Infrared Modalities for Improved Aerial Vehicle Detection

Data from aerial sensing platforms has long been critical for Intelligence, Surveillance, and Reconnaissance (ISR) for the US military. Traditionally, the computer vision algorithms used to analyze the visual data from these platforms take in a single modality such as electro-optical (EO) data as RGB images and return meaningful details about what is contained in the image. An increasing emphasis on multimodal learning now encourages the synthesis of visible imagery with infrared (IR) to boost performance. The expected synergy between EO and IR data is a result of the inherent differences in what they tell about a scene. This paper proposes a simple and cost-effective approach to outperform unimodal methods for the object detection of vehicles from an aerial perspective. Using the public dataset VisDrone-DroneVehicle, we compare unimodal results to that of a basic decision-level approach we refer to as the "switch approach". The switch approach operates under the assumption that EO imagery may be advantageous in well-lit scenes whereas IR imagery will be advantageous in poorly lit scenes. This approach utilizes specialized unimodal models and a brightness threshold to decide which model to use for each scene. We then provide an augmented switch approach in which both modalities are used without regard for the scene brightness. Ultimately, the switch approach outperforms unimodal methods, achieving a mAP of 0.759 compared to the IR performance of 0.755. Furthermore, the initial results of the augmented switch approach outperform the EO performance with a mAP of 0.692 compared to 0.685 but fail to compete with the ordinary switch and IR-only approaches. These results advocate for using both modalities rather than one or the other and suggest that scene lighting could be a relevant factor in successful decision-level fusion approaches.