Multimodal object detection: an architecture using feature-level fusion and deep learning

Abstract Object detection is one of the most fundamental problems to tackle in the computer vision research area. Recent advances in multimodal data streams and deep learning architectures have prompted a fast growth in the field of multimodal learning, which brings several advantages over single-modality approaches for object detection, such as improved accuracy, robustness to noise and ambiguity, handling of complex scenarios and adaptability to diverse data. Some of the biggest challenges when implementing a multimodal learning approach are the selection of the fusion strategy, design of processing architecture, modality alignment/synchronization and interpretability of such high-dimensional representations. To address this challenge, we propose a feature-level fusion architecture for object detection based on extracting YOLO features from images, spectral and rhythm features from sound using Mel-frequency cepstral coefficients, and general descriptors from radar modalities that, after timestamp and homography transformation matrix alignment, are combined with an attention mechanism into a single classification network. Preliminary experiments indicate that the proposed architecture can constitute itself as a base pipeline for several different multimodal object detection tasks in real-world applications.