Bayesian optimization across the spectrum of knowledge : enhancing efficiency through beliefs, information and assumptions

Bayesian Optimization has emerged as a crucial technique for optimizing costly, black-box functions where each evaluation comes at a high cost, such as in scientific experiments, and machine learning hyperparameter optimization. By combining probabilistic modeling with sequential decision-making, Bayesian Optimization achieves efficient exploration, guiding the search toward optimal parameters with minimal data. However, real-world applications present three main challenges: leveraging expert knowledge, ensuring accurate model assumptions, and managing high-dimensional search spaces.This thesis addresses these challenges by advancing Bayesian Optimization in three key areas. First, it develops methods to incorporate practitioner insights directly into the optimization process, using domain expert knowledge to guide the search more efficiently and reduce the need for extensive evaluations. Second, it proposes techniques for dynamically validating and adapting model assumptions, enabling the Gaussian Process surrogates commonly used in Bayesian Optimization to align more closely with the complexities of real-world objective functions. Finally, this work introduces adaptive strategies for high-dimensional optimization, allowing Bayesian Optimization to focus on relevant subspaces and improve sample efficiency in vast parameter spaces, thereby mitigating the "Curse of Dimensionality."These contributions collectively enhance Bayesian Optimization’s robustness, adaptability, and efficiency, positioning it as a more powerful tool for sample-efficient optimization in complex, resource-intensive scenarios. By demonstrating these improvements through theoretical insights and empirical evaluations, this thesis establishes a pathway for more effective Bayesian Optimization in diverse, real-world applications where data is sparse and costly to obtain.