Data Conflicts-Guided Interleaved Thread Scheduling for Flaky Test Detection in Multithreaded Programs

Flaky tests can non-deterministically pass or fail on the same version of code. Their occurrence prevents using test results to determine if there are bugs in the program, thereby reducing the credibility of software testing. Concurrency is a main cause of flaky tests, where different concurrent thread interleaving may lead to uncertain program execution results. Essentially, inconsistent test execution results are more likely to occur when threads with data dependencies are interleaved. But the existing work scheduled threads randomly with no guide during flaky test detection, resulting in a vast interleaving space and low efficiency. To address this issue, we propose a data conflict-guided flaky test detection approach, which prioritizes threads with data dependency for scheduling to try to alter the test results, thereby detecting flaky tests more effectively. In more detail, the dynamic execution trace of test cases is tracked in multithreaded programs. Then, by analyzing read-write operations in the trace, data conflicts are identified. On this basis, a Bayesian Network is put forward to evaluate the degree of impact of data conflicts on test flakiness, which implies potential flaky risk. That is, the greater the flaky risk, the more likely the data conflict is to cause a flaky test. So, the scheduling strategy prioritizes threads with data conflict pairs that have a significant impact on test flakiness, to trigger test flakiness as soon as possible. To verify the effectiveness of our approach, we conduct experiments on multithreaded Java programs. The experimental results show that our approach can discover the flakiness of tests within an average of 118 seconds, and the accuracy in detecting flaky tests can reach 75 %.