Can test smells affect the occurrence of flaky tests? An investigative study

Ensuring test code quality is essential for maintaining reliable and maintainable software. However, test code is often neglected in software projects, leading to the presence of test smells—poor test design practices that can degrade maintainability and obscure potential issues. While test smells do not necessarily indicate defects, they can introduce complexity, making tests harder to understand and maintain, ultimately affecting their reliability. Previous studies suggest that refactoring test smells may help mitigate flaky tests, which are tests that produce inconsistent results across different executions without changes to the test or production code. Flaky tests undermine confidence in automated testing, as they can conceal real defects and increase debugging efforts.Despite the potential link between test smells and flaky tests, limited research has explored this relationship in depth. This study investigates whether test smells influence the occurrence of flaky tests by analyzing 82 Java projects containing at least one confirmed flaky test. We applied automated detection techniques to identify test smells and conducted a manual analysis at the method level to examine their co-occurrence with flaky tests.The results indicate that specific test smells are more frequently associated with flaky tests. Notably, Lazy Test, Sleepy Test, Assertion Roulette, and Exception Catching Throwing were among the test smells most linked to flaky test occurrences. Our findings suggest that Lazy Test is particularly associated with Order Dependency flaky tests, while Sleepy Test is linked to Async Wait flaky tests. Additionally, we observed that test smell co-occurrence is common in methods affected by flaky tests, with Magic Number Test and Assertion Roulette being the most frequent combination, appearing together in seven instances.As a key contribution, this study presents a catalog of test smells associated with flaky tests, offering insights into which test smells may contribute to test flakiness. By understanding these relationships, developers can adopt better test design practices to improve test reliability and reduce flakiness in automated test suites.