Combinatorial Testing of Engineered Systems With Hard‐to‐Change Factors

ABSTRACT The testing of engineered systems is a complex task, which often requires a test engineer to construct a suite of test cases in order to investigate system behavior, with the primary purpose of this endeavor being to detect errors in the system. Test engineers often must strategically construct their suite of test cases due to budgetary and other operational constraints. Combinatorial testing is an effective and efficient method to construct such test suites. The mathematical object typically used for construction is a covering array, where the columns of the array correspond to the factors of the system under test (SUT). In some instances, there may be a subset of factors that are expensive or hard to change. In the statistical design of experiments setting, split‐plot designs are used when hard‐to‐change factors are present to accomodate randomization constraints. We adopt a similar structural approach, focusing on addressing practical implementation constraints such as reducing the number of resets of hard‐to‐change factors, which can be costly or time‐consuming. Borrowing the concept of split‐plot designs from the design of experiments, we extend the methodology of combinatorial testing by presenting a method for constructing covering arrays with split‐plot structure. We then demonstrate the efficiency of our method in a case study involving the testing of an implementation of XGBoost.