Improving Fault Localization with External Oracle by Using Counterfactual Execution

We present Flex , a new approach to improve fault localization with external oracles. Spectrum-based fault localization techniques estimate suspicious statements based on the execution trace of the test suite. State-of-the-art techniques rely on test oracles that internally exist in the program. However, programs often have external oracles that observe their behavior from outside. This in turn hinders fine-grained and accurate estimation of suspicious statements in practice because the correctness of each execution can only be observed at termination. In this article, we aim to address this problem by observing counterfactual execution traces, which enable fine-grained estimation even without precise internal oracles. We observe two types of counterfactual scenarios related to different types of test cases: When the branch condition is set to a Boolean constant, (1) if most of the passing test cases still pass, we consider the newly executed statements in the branch statement as unrelated to the failure; (2) if failing test case still fails, we also consider the originally executed statements as unrelated to the failure. We evaluated the performance on widely used C and Java programs. Flex improves the accuracy of state-of-the-art SBFL techniques on C and Java programs by 24% and 22% on average, respectively.