An empirical evaluation of white-box and black-box test case prioritization techniques in CPSs modeled in Simulink

Abstract MATLAB/Simulink is the leading tool for simulating complex Cyber-Physical Systems (CPSs). The simulation models of complex CPSs are typically compute intensive, and the execution of test cases is long. Furthermore, the execution of test cases is typically triggered several times at different “in-the-Loop” test levels (i.e., Model, Software and Hardware-in-the-Loop). Therefore, test optimization techniques, such as test case prioritization, are paramount when testing these systems. In this paper, we present what to the best of our knowledge is the first empirical study on test case prioritization techniques for Simulink models by comparing the performance of white-box and black-box test case prioritization techniques. We assess traditional test case prioritization techniques, and we also propose new approaches for use in the context of Simulink models. We empirically compared 11 test case prioritization techniques using six Simulink models of different sizes and complexities. When comparing white-box against black-box test case prioritization techniques, we found that in general, white-box techniques were slightly better than black-box ones. In the context of white-box test case prioritization, the total greedy approach outperformed the additional greedy strategy in models with higher block interactions (>10) and connections (>200), metrics that better capture system coupling than raw block count. As for the test case prioritization time, black-box techniques were faster, although total greedy techniques were fast enough to be used in practice.