An Environment Adaptation Agent of Reinforcement Learning in Continuous Integration Test Case Prioritization

Continuous Integration (CI) is a fundamental practice in modern software development. It enables early fault detection through regression testing, where Test Case Prioritization (TCP) plays a crucial role in improving detection efficiency. Reinforcement Learning-based TCP (RL-TCP) has shown promise in CI scenarios, but its performance often fluctuates due to CI’s dynamic nature. Existing solutions address this by assigning additional rewards or periodically retraining agents. However, these methods either risk over-adjusting strategies due to excessive reliance on additional rewards or fail to respond promptly to changes due to fixed retraining intervals. We propose a novel Environment Adaptation Agent-based RL-TCP method (EAA) that addresses these challenges through a dual mechanism. EAA detects significant environmental changes by analyzing fluctuations in prioritization effectiveness. When a change is detected, it assigns targeted rewards to test cases. EAA also refines the agent’s gradient update so that environmental dynamics are better incorporated into retraining. This enables agents to swiftly adapt while retaining learned prioritization knowledge. Evaluations on 12 real-world industrial datasets show that EAA improves the NAPFD metric by 4.7–24.79% and reduces the average TTF by 35.85–50.37 positions compared to state-of-the-art RL-TCP methods. Additionally, EAA significantly reduces occurrences of NAPFD equal to zero, effectively mitigating prioritization instability.