An Exploratory Study on the Impact of Change-Proneness as a Metric in Black-Box Test Suite Minimization

Black-box Test Suite Minimization (TSM) aims to remove redundant test cases from a test suite while retaining its fault detection capability without analyzing the production code. This makes it particularly efficient to be used in industry projects. These techniques utilize metrics such as test history, commit complexity or test code diversity (or similarity) to guide test case selection. Similarly, change-proneness (CP) can also guide to successful test case selection as it indicates the likelihood of having faults. We propose CP as a metric for TSM and implement into an approach, Change-proneness based Test suite Minimization (CTM), where relationships between test cases and their depending classes are measured using CP values. CTM first calculates class-level CP, followed by identifying the dependency between the test cases and classes. Then the association between test cases and their related classes are calculated using various statistical measures such as geometric mean etc. Finally test cases with the highest association values are selected. To demonstrate the effectiveness and efficiency of CP as a TSM metric, we compared CTM with state of the art, AST-based Test case Minimizer (ATM) on 15 Java projects with 617 versions. The experimental results demonstrate that CTM achieves the same average fault detection accuracy as ATM (0.67) while reducing execution time by 114.5 folds.