Runtime Monitoring Abnormalities in Object Detection With Spatial and Temporal Abstractions

ABSTRACT Background Object detection modules are essential functionalities for any autonomous vehicle. However, the performance of such modules implemented using deep neural networks can be unreliable in many cases, which raises the necessity to filter the abnormal outputs for safety considerations. Aims This article aims to develop a logical framework for filtering potentially erroneous object detection results. Materials & Methods Concretely, we consider two types of abstraction, namely spatial abstraction and temporal abstraction, based on the data labels from the training dataset of object detectors, and temporal consistency between a sequence of images. Operated on the training dataset, the construction of spatial abstraction iterates each input, aggregates region‐wise information over its associated labels, and stores the object abstraction. The abstraction is adopted to filter static and spatial abnormalities. The temporal abstraction builds an abstract transformer for a relaxed tracking algorithm. Elements being associated together by the abstract transformer can be checked against consistency over their original values. The abstraction helps to monitor temporal abnormality in consecutive frames. We have implemented the overall framework and validated it using publicly available datasets and open‐source object detectors. Results The implemented framework successfully identified most of static/spatial and temporal abnormalities in object detection outputs. Validation on public datasets confirmed the effectiveness of the abstraction‐based approach in filtering unreliable detections. Discussion The results demonstrate that logical abstractions derived from training data labels and temporal sequences provide a viable mechanism for monitoring the reliability of deep neural network‐based object detectors. Conclusion Abstraction‐based monitoring presents a robust and logical framework for enhancing the reliability of object detectors by filtering abnormal detection results.