Diversity Learning by Minimizing Construction Error of Ensemble Loss for Bayesian Optimization

Ensemble learning has been shown to achieve state-of-the-art performance in various applications by integrating multiple diverse base models. Most automated machine learning (AutoML) tools utilize Bayesian Optimization (BO) with post-hoc selection to construct ensembles. However, traditional BO focuses on identifying a single model with optimal performance, disregarding diversity among models, which restricts the potential effectiveness of an ensemble. To address this issue, some approaches have introduced an additional diversity surrogate to predict pair-wise diversity during BO iterations. In the literature, diversity measures, such as Euclidean distance, defined heuristically, or measures derived from the upper bound of ensemble loss for specific cases have been used to train diversity surrogates. To enhance generality and effectiveness, this paper presents the Diversity Learning Bayesian Optimization (DLBO) approach where a diversity surrogate is learned by minimizing the construction error of ensemble loss. The DLBO method eliminates the need to predefine a specific diversity measure and is adaptable to any type of loss function and combinator for ensemble. The resulting diversity surrogate can be also meaningfully integrated with the performance surrogate to predict ensemble loss more precisely, supporting the development of an effective acquisition function for optimal ensemble searching. Evaluation results demonstrate that the proposed method is a more effective strategy for constructing high-quality ensemble models compared to state-of-the-art methods.