Predicting the Root Cause of Flaky Tests Based on Test Smells

Flaky tests refer to test cases that exhibit inconsistent behaviors across multiple executions, potentially passing or failing unpredictably. They are frequently associated with suboptimal design practices that testers may utilize when crafting test cases, which undermine the quality of software testing. So, identifying the root causes of flaky tests is crucial for fixing them. Currently, inspired by the success of the Large Language Models (LLMs), researchers leverage the pre-trained language model to embed flaky test code as vectors and predict its root cause category based on vector similarity measures. However, such code embeddings generated by LLM mainly focus on capturing general semantic features but lack sufficient comprehension of the behavioral patterns involved in test scenarios, resulting in poor root cause identification. Test smells, which reflect poor coding practices or habits when writing test cases, provide complementary information in the root cause identification of test flakiness. Therefore, this paper proposes a root cause identification method for flaky tests based on test smells. Test smells are used to abstract and express behavioral patterns of test codes, and general semantic features extracted by vector embeddings to enhance the feature representation of flaky tests. Furthermore, to capture the complex nonlinear relationships between test smell features and code embeddings, a Feedforward Neural Network is constructed to categorize the root cause of test flakiness. To validate the effectiveness of our method, we performed evaluations on a dataset consisting of 451 Java flaky test cases. The experimental results indicate that our method achieves an F1-score of 80%, which is 7% higher than that of the baseline model that does not incorporate test smells.