Context-Aware Sequential Ranking for Dynamic Test Case Prioritization in Continuous Integration

Regression testing in continuous integration (CI) helps preserve delivered software value by exposing regressions early, but limited feedback windows make test order consequential. This study evaluates a leakage-safe dynamic test case prioritization framework that reconstructs job–build–commit provenance, combines strictly prior-build history with change–test similarity, applies class-conditional conformal calibration, and evaluates an operator-level deep Q-network (DQN). The primary held-out evaluation contains 273 failure-bearing jobs and 45,774 job–test rows from four Java projects. Pretrained history–semantic fusion achieved an equal-project macro failing-test-entity APFD surrogate (FTE-APFD surrogate) of 0.8840, compared with 0.8689 for history-only and 0.7023 for pretrained semantic-only ranking; the fusion–history difference did not survive Holm correction. A train-only TF–IDF control reached 0.7502 for semantic-only ranking and 0.8871 when fused with history in a seed-averaged diagnostic, so the benchmark does not establish unique superiority of the pretrained representation. A protocol-aligned RETECS reimplementation achieved 0.8296 ± 0.0111 FTE-APFD surrogate across five seeds. Adaptive conformal calibration reduced the candidate fraction from 0.3171 to 0.2598 while retaining 0.9058 failing-test coverage. The candidate-informed gated DQN remained below strong deterministic fusion rankings at 10%, 25%, and 50% budgets; five-seed, short-budget, hyperparameter, pre-execution-budget, and project-exclusion sensitivities did not establish a consistent DQN advantage. An additional three-project operational analysis containing 444 passing test jobs showed high failing-row coverage but strongly project-dependent candidate burden. Overall, the evidence favors strong leakage-safe deterministic rankings with explicit uncertainty control, while the evaluated reinforcement-learning design remains a local negative result rather than a general conclusion about reinforcement learning.

Authors

Institutions

Publication Details

Journal
Mathematics
Published
2026-09-09
DOI
https://doi.org/10.3390/math14183276
Primary Topic
Software Testing and Debugging Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Context-Aware Sequential Ranking for Dynamic Test Case Prioritization in Continuous Integration

Umut Altınışık
Mathematics
Software Testing and Debugging Techniques
article

Context-Aware Sequential Ranking for Dynamic Test Case Prioritization in Continuous Integration

Umut Altınışık
article en

Abstract

Regression testing in continuous integration (CI) helps preserve delivered software value by exposing regressions early, but limited feedback windows make test order consequential. This study evaluates a leakage-safe dynamic test case prioritization framework that reconstructs job–build–commit provenance, combines strictly prior-build history with change–test similarity, applies class-conditional conformal calibration, and evaluates an operator-level deep Q-network (DQN). The primary held-out evaluation contains 273 failure-bearing jobs and 45,774 job–test rows from four Java projects. Pretrained history–semantic fusion achieved an equal-project macro failing-test-entity APFD surrogate (FTE-APFD surrogate) of 0.8840, compared with 0.8689 for history-only and 0.7023 for pretrained semantic-only ranking; the fusion–history difference did not survive Holm correction. A train-only TF–IDF control reached 0.7502 for semantic-only ranking and 0.8871 when fused with history in a seed-averaged diagnostic, so the benchmark does not establish unique superiority of the pretrained representation. A protocol-aligned RETECS reimplementation achieved 0.8296 ± 0.0111 FTE-APFD surrogate across five seeds. Adaptive conformal calibration reduced the candidate fraction from 0.3171 to 0.2598 while retaining 0.9058 failing-test coverage. The candidate-informed gated DQN remained below strong deterministic fusion rankings at 10%, 25%, and 50% budgets; five-seed, short-budget, hyperparameter, pre-execution-budget, and project-exclusion sensitivities did not establish a consistent DQN advantage. An additional three-project operational analysis containing 444 passing test jobs showed high failing-row coverage but strongly project-dependent candidate burden. Overall, the evidence favors strong leakage-safe deterministic rankings with explicit uncertainty control, while the evaluated reinforcement-learning design remains a local negative result rather than a general conclusion about reinforcement learning.

MathematicsVol. 14(18)
Kocaeli Üniversitesi (TR)
Decent work and economic growth
Openalex Percentile: Top 5%
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.