Evaluating the performance of NSGA-II and NSGA-III on Product Line Architecture Design

Product Line Architecture (PLA) design can be modeled as an optimization problem to be solved with search-based algorithms. PLA design optimization has successfully been done using the Non-Dominated Sorting Genetic Algorithm II (NSGA-II) in scenarios involving up to three objectives, which involve software metrics for properties such as feature modularization, PLA extensibility, and cohesion. As many of these properties may be desired in a PLA at the same time, more than three objectives might need to be optimized simultaneously. The Non-Dominated Sorting Genetic Algorithm III (NSGA-III) was designed to solve problems impacted by more than three objectives, named many-objective problems, so it might suit this need. However, NSGA-III has not yet been applied in the context of PLA design. In this sense, this study aims to compare the performance of NSGA-II and NSGA-III for PLA design to uncover which algorithm best fits this problem. To accomplish this goal, we implemented a specialized version of NSGA-III and then ran experiments using both algorithms to optimize eight PLAs with three, four, and five objectives. We evaluate the algorithms’ performance via quality indicators commonly used in search-based software engineering. The empirical results point out that: (i) NSGA-III had a slightly better performance than NSGA-II when optimizing four or five objectives in the context of our study; (ii) NSGA-II was the best or the algorithms tied when the PLA given as input is easier to optimize due to reduced solution space.