Collecting Feature Models from the Literature: A Comprehensive Dataset for Benchmarking

Feature models are widely used for specifying the valid configurations of product lines. Many automated analyses on feature models have been considered, but they often depend on computationally complex algorithms (e.g., solving satisfiability problems). To identify and develop efficient reasoning engines, it is necessary to compare their performance on practically relevant feature models. However, empirical evaluations on feature-model analysis often suffer from the limitations of available feature-model datasets in terms of transferability. A major problem is the accessibility of relevant feature models as they are scattered over numerous publications. In this work, we perform a literature survey on empirical evaluations that target the performance of feature-model analyses to examine common evaluation practices and collect feature models for future evaluations. Furthermore, we examine the suitability of the derived collection for benchmarking performance. To improve accessibility, we provide a repository including all 2,518 identified feature models from 13 application domains, such as system software.