Specification-Based Testing of the Image-Recognition Performance of Automated Driving Systems

Automated driving systems (ADSs) are complex entities comprising numerous components, and traditional testing methods often struggle to ensure their safety, primarily due to the diversity driving environments. Interestingly, deep neural networks (DNNs) have proven effective for object detection in these settings. The safety of object detection in ADSs depends on the position of the detected objects and the specifications that guide the system’s response to them. Consequently, testing the object-detection process in ADSs must be grounded in these specifications. However, current specifications are informal regarding object locations and inadequate for object-detection testing. To address this issue, this article first introduces the bounding box specification language (BBSL), a framework capable of mathematically articulating the specifications for object and event detection and responses. Subsequently, we propose a specification-based testing approach for the object-detection process in ADS using BBSL. Remarkably, BBSL can formally delineate the positions of objects within the driving environment. Furthermore, our proposed approach can identify safety-critical defects that conventional tests, which focus solely on performance evaluation, might overlook. Furthermore, we propose two sets of test criteria. The first set reflects the diversity of object positions and sizes within an image, while the second set includes coverage metrics that determine whether the test cases cover all conditions outlined by the BBSL specifications. Overall, our contributions facilitate the implementation of specification-based testing for object-detection systems using DNNs, a challenge previously considered formidable.