Adaptive Acquisition in Bayesian Optimization with Agnostic Ensembles

Bayesian Optimization (BO) is a popular black-box optimization method consisting of a surrogate model, typically a probabilistic model such as a Gaussian Process (GP) and an Acquisition function (AF). Effective selection of these functions has a strong impact on the optimization process. Existing ensemble-based methods for AF selection operate under the assumption that an "optimal" AF exists in the pool of considered AFs, and the method attempts to find the best AF. In this work, we operate in an agnostic setting and consider the optimal AF to be one which minimizes joint risk over all AFs considered. This allows us to treat the joint risk as a random process and perform Bayesian inference on the posterior. We empirically demonstrate the effectiveness of this method and provide theoretical bounds on the regret.