Bayesian Approaches for Robust and Efficient Sequential Decision-Making

In many situations, the solution to a problem is not found in one fell swoop, but iteratively, by making a series of decisions to ultimately achieve the desired goal. The formal framework for this process is referred to as sequential decision-making. Here, the next step is determined by taking into account the available information and the possible consequences of the decision. Methods from the field of sequential decision-making are used in chemistry, robotics or engineering, but their application is still limited due to open challenges. This work deals with two of them: sample efficiency and robustness. Sample efficiency is crucial when the data required for decision-making is costly, which is the case for many real-world problems. Moreover, in these cases, the solution to the sequential decision-making problem is often found in a setting that differs from the final application, so additional robustness is required. We focus on finding adversarially robust solutions, i.e., solutions that are optimal even under the worst application conditions - a typical but conservative robustness assumption. In this work, we develop sample-efficient and adversarially robust algorithms for three subdomains of sequential decision-making: Bayesian Optimization, Active Learning, and Reinforcement Learning. In all algorithms, we achieve high sample efficiency through Bayesian methods that efficiently combine complex expert knowledge with data and, by exploiting their inherent uncertainty estimates, enable information-based methods that are known for their high sample efficiency. Furthermore, in order to find solutions that are adversarially robust at the time of application, we already include both the possible application scenarios and the robustness requirements during the sampling process. The results obtained extend the state of the art considerably. In Bayesian Optimization, we search for the optimum of a costly-to-evaluate black-box function that is robust to a worst-case realization of parameters that are uncontrollable at the time of application. Current algorithms focus either on sample efficiency or on robustness to adversarial attacks, but not on these two problems simultaneously. In Active Learning we deal with a specific problem that has arisen in practice: We learn a probabilistic binary classifier and focus on finding the worst possible sublevel set of the class probability function under the possible application conditions. Competing approaches are either sample-efficient Bayesian approaches to level set estimation, which thus lack the robustness component; or they focus on general Active Learning problems with corruptions during the learning process, and thus do not fit the specific setup. In Reinforcement Learning, we find an optimal sequence of actions to reach a certain final state that also works under the worst-case realization of the environment. Therefore, we assume the environment to be represented by a discrete set of oracles that all match the ground truth environment for different application cases. In the case of multiple oracles, the current state-of-the-art methods try to identify the correct model, while optimizing a risk measure over the oracles. Our approach differs as follows: we restrict ourselves to finding a worst-case robust solution, so we do not devote resources to the identification of the correct oracle. Additionally, we further increase the sample efficiency by exploiting knowledge about the variation of the samples gained from the oracles. For all mentioned subdomains of sequential decision-making we derive adversarially robust and sample-efficient algorithms. Their superior performance over the state of the art is demonstrated empirically on two kinds of problems. First, on variants of traditional benchmark problems from the individual subdomains that consider the robustness requirement. Here, we show that our approaches are both robust (as they find the robust optima), and sample-efficient as they find them faster than competing robust approaches. Secondly, we demonstrate the usefulness of our approaches on engineering problems from the real-life applications that originally motivated our work. In this way, we can show that our approaches are not just theoretical, but really relevant in practice.