Closing the Optimization and Generalization Gap in Deep Learning with A Multi-Objective Evolutionary Algorithm

SGD dominates deep learning despite the demonstrated advantages of evolutionary algorithms (EAs) for non-convex optimization. In this paper we introduce MO-DL-HEA, a multi-objective hybrid EA that outperforms SGD in real-world tasks while addressing the generalization gap through simultaneous training and validation optimization, demonstrating superior alternatives to SGD for deep learning. Our experiments across multiple architectures and datasets consistently show MO-DL-HEA achieving better generalization with comparable or lower training loss than SGD. The algorithm's two-tier selection mechanism—using validation loss for parent selection and generalization gap for survivor selection—effectively balances exploration of novel solutions with refinement of near-optimal ones, driving the population toward models that generalize well to unseen data.