Automl Framework for Ensembling: Split Validation Sets for Coefficient and Structure Selection

We propose AutoESSV, a versatile method for choosing ensemble strategies by utilizing a split validation set. This approach incorporates a second validation set, enabling the ensemble meta-learner to identify the most suitable ensemble selection strategy for the data while preserving sufficient data for internal model validation. The first validation set focuses on optimizing model weights, facilitating the selection of an ideal ensemble strategy and structure without risking data leakage, thanks to the second validation set. To evaluate our novel ensemble selection method, we benchmark its performance against the state-of-the-art AutoSklearn framework using sixteen classification and regression datasets from the UCI Repository.