An ISO 26262-Derived Evaluation Methodology for Automated Fault Injection Test Case Generators

Fault Injection (FI) is a well-established method to assess the effect of failures within elements of a system under test. Where FI test cases can be executed automatically, such as in simulation-based or Hardware-in-the-Loop (HiL) FI, numerous test case generators (TCGs) have been proposed that aim to uncover more ‘critical’ test cases or to accomplish this using fewer resources. However, the evaluations of these approaches do not allow for direct comparisons: Experiments are often not reproducible, metrics are commonly specific to use cases, and key properties, such as test case distribution, are often not captured. Further, the authors are not aware of any suitable comparison frameworks. Hence, to support practitioners in selecting the most suitable FI TCG for their use case, test setup, and individual goal, this work introduces a set of use case-independent metrics derived from the ISO 26262 safety standard and identifies how these metrics can be applied and analyzed to capture decisive characteristics such as the distribution, criticality, and coverage of generated test cases. We incorporate these metrics and analyses in a start-to-finish methodology and provide their implementation as an open-source tool to effectively and reproducibly evaluate TCGs for automated Fl. The evaluation methodology is assessed in a case study with an industry-oriented cyber-physical system, demonstrating its ability to support practitioners in making an informed decision about the TCG providing the most ap-propriate balance of coverage and efficiency for their particular use case.