Fuse It or Lose It? Analyzing the Effects of Sensor Diversity on Multimodal Ensembles for Autonomous Vehicle Perception

Autonomous vehicles (AVs) can operate in complex environments that require multiple types of sensors in their perception approaches. However, a single sensing configuration is not tenable in all environments, and adapting the perception approach to different domains is a challenging task. Both sensor fusion and ensemble learning methods utilize the diversity of multimodal sensor data (e.g., cameras, radar, lidar) to increase perception performance under challenging sensing conditions. In this paper, we conduct the first analysis examining how sensor diversity can impact performance across different AV perception tasks. We propose ensembles of multimodal models that leverage diversity and assess their ability to address the challenge of adaptation in AV perception. We introduce a novel, model-agnostic framework, <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DivFusE</b>, that identifies the level of diversity that individual sensor modalities contribute to perception ensembles and employs adaptation based on this diversity. We benchmark our approach on five public datasets containing semantic segmentation and object detection tasks, highlighting scenarios where our proposed framework can improve perception performance by 70% (<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fuse it</b>) and scenarios where fusion can reduce perception performance (<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">lose it</b>). Our findings indicate that the performance improvements of using multimodal ensembles is greater on objection detection compared to semantic segmentation, and that sensing conditions with higher (lower) levels of diversity further increase (decrease) performance. Ultimately, the analysis in this work highlights scenarios where these multimodal ensembles should and should not be deployed and can be used to develop more adaptive multimodal sensor fusion systems.