Understanding the potentially confounding effect of test suite size in test effectiveness evaluation

Background: Code coverage and mutation score serve as pivotal test effectiveness metrics used to assess a test suite's ability to uncover actual defects. However, prior research has produced inconsistent or even conflicting findings regarding their correlation with defect detection capability, particularly concerning the impact of test suite size. Problem: The extent of the potentially confounding effect of test suite size in test effectiveness evaluation context is not clear, nor is the method to remove the potentially confounding effect, or the influence of this removal on the performance of test suite optimization. Objective: Our goal is to deeply understand how test suite size affects the true relationship between test effectiveness metrics and a test suite's ability to detect actual defects. Method: We first employ statistical methods to examine the extent of the potentially confounding effect of test suite size in the context of test effectiveness evaluation. After that, we propose a linear regression-based method to remove the potentially confounding effect of test suite size. Finally, we empirically explore the impact of this removal method on test suite optimization. Result: Our experimental results, based on the Defects4J defect dataset, uncovers that: (1) the confounding effect of test suite size on the associations between test effectiveness metrics and defect detection capability in general exists; (2) the proposed linear regression-based method can effectively remove the confounding effect; and (3) after removing the confounding effect, mutation score demonstrates superior effectiveness in predicting test suite effectiveness, while statement coverage is the least effective metric. Furthermore, both coverage-based and mutation-based test suite reduction exhibit enhanced cost-effectiveness in defect detection, and there is a marginal improvement in the speed of defect detection for coverage-based test case prioritization. Conclusion: When using test effectiveness metrics to assess test suite effectiveness, it is crucial to eliminate the influence of test suite size.