Beyond Test Flakiness: A Manifesto for a Holistic Approach to Test Suite Health

Large numbers of flaky tests are a sure sign of a dysfunctional, or “unhealthy”, test suite. In this paper, we identify several further indicators of unhealthy test suites, arguing that trade-offs exist among some indicators, with others complementing one another. We encourage researchers and practitioners not to stop at test flakiness-or any individual metric of “good” or “bad” tests-and instead focus on developing and adopting techniques and tools that holistically address test suite health. For more information, see https://philmcminn.com/test-suite-health.