Multi-Layer Self-Assessment with Filtering for 3D Object Detection in Autonomous Vehicles

Reliable detection of road users is critical to the safety of automated driving systems. While object detectors based on deep neural networks are widely used for this purpose, they remain susceptible to errors that could compromise safety. A promising strategy to mitigate these risks involves run-time perception monitoring mechanisms, commonly referred to in the literature as self-assessment or introspection. Current research in this area predominantly addresses anomaly detection, or monitoring camera-based 2D object detection, with insufficient focus on in-distribution errors and 3D object detection. Additionally, existing 2D studies often monitor activation patterns from the final layers of the network backbone, overlooking earlier activations that preserve higher spatial resolution. Yet, high-resolution early-layer activations can be valuable for detecting errors with sparse 3D point clouds. We also argue that not all objects in a scene should equally influence frame-level error detection, a factor often neglected in current methods. To address these gaps, we propose a novel self-assessment mechanism for 3D object detection that leverages activation patterns from multiple network layers. This mechanism employs spatial filtering to focus the model within an area of interest in the close vicinity of the ego vehicle. Additionally, it utilises an object filtering mechanism, which specifically targets the missed objects by excluding the points in those already detected. We evaluate our method using widely recognised object detectors and public datasets. Additionally, we demonstrate its robustness under domain shifts with real-world LiDAR data collected on motorways in diverse weather conditions. Results show the proposed mechanism provides 6% AUROC improvement over last-layer activation methods with spatial filtering on the NuScenes dataset. It also demonstrates a superior ability to transfer knowledge under domain shifts. Code is available at https://github.com/yatbazhakan/multi-layer-introspection .