Analyzing Test Case Prioritization approaches With Performance and Speed Trade-Offs in CI Workflows

The software testing process is an important activity in software development that helps ensure applications are working properly and meet quality standards. Among testing methodologies, regression testing plays a pivotal role in ensuring that recent changes do not introduce new defects in previous parts of the software. However, as software applications increase in size and complexity, executing all test cases becomes increasingly difficult and time-consuming. Test case prioritization effectively addresses this challenge by identifying and selecting the most likely test cases to fail, optimizing fault detection within the available time constraints. This study provides a comparative analysis of three state-of-the-art test case prioritization techniques: COLEMAN (Combinatorial VOlatiLE Multi-Armed BANdit), Reinforced Test Case Selection (RETECS), and DeepOrder. Each model uses a different approach to prioritize test cases: combinatorial optimization, reinforcement learning and deep learning respectively. The study compares the model’s ability to detect faults earlier in the testing process using various metrics such as Normalized APFD (NAPFD) and prioritization time.