Feature-oriented Test Case Prioritization Strategies: An Evaluation for Highly Configurable Systems

Test Case Prioritization (TCP) has proved to be beneficial when time constraints exist. During TCP, most relevant test cases are executed first in order to maximize failure detection and produce rapid feedback. There are several TCP approaches in the literature, but most of them do not consider particularities of Highly-Configurable Systems (HCSs), namely the notion of features. Existing work is usually based on artifacts (e.g., feature models) that can be outdated or are not available. Some code-oriented approaches usually employ dynamic analysis or require a failure-history, which can be costly. Furthermore, they present limitations when new variants and test cases are added during the system evolution. To address such limitations, we propose and evaluate three feature-oriented TCP strategies: (i) FeatChgHist: prioritizes test cases associated to features that often changed over commits; (ii) FeatChgCommit: prioritizes test cases associated to features that changed more in the current commit; and (iii) FeatCov: prioritizes test cases that cover more features simultaneously. Features are the building blocks of HCSs that are used for communication and design purposes. Then the basis to introduce these strategies is the fact that test cases associated to a great number of features, or to features that changed often over commits, are more fault-prone. In the evaluation, conducted with a real open-source HCS, FeatChgHist and FeatChgCommit show better performance regarding failure detection than FeatCov, and also than the changed-file-oriented strategies, used as baseline. Moreover, combining test selection and prioritization allows a reduced test set without decreasing significantly early failure detection.