Validation-based Decision Making in Data-driven Evolutionary Computation: A Case Study in Multi-objective Feature Selection

Overfitting occurs when a model captures noise and incorrect patterns, reducing its accuracy. This challenge extends to data-driven evolutionary optimization, such as multi-objective feature selection, where relying on training data for decision-making can degrade test set performance. Inspired by machine learning practices, this paper proposes using validation data for decision-making to reduce overfitting in multi-objective feature selection, focusing on classification error rate and the number of selected features as objectives. Although algorithm-independent, the framework is demonstrated using established multi-objective optimization algorithms. New metrics compare Pareto fronts from training- and validation-based approaches. Experiments on fourteen datasets reveal that validation-based decision-making significantly outperforms training-based methods, particularly by reducing the number of selected features while maintaining effectiveness.