Theory and Applications of Bayesian Optimization

Economics examines various optimization problems, such as cost minimization and utility maximization. In many optimization problems, the maximum (or minimum) value of a differentiable continuous function defined on a bounded closed set satisfies the Kuhn-Tucker condition. However, in many cases, an analytical solution either does not exist or is difficult to obtain. For such problems, numerical methods such as Newton’s method, gradient descent, conjugate gradient, and the simplex method are used to find solutions 1). However, what is obtained is a local optimum, and it is not necessarily a global optimum. In the case of problems with one or two variables, grid search or random search is also used, but in general, the computation takes a long time. Even with improvements in computational performance, these methods have their limitations. On the other hand, in the field of machine learning, Bayesian models are used to adjust hyperparameters. This method, called Bayesian optimization, can also be applied to function maximization problems. This method not only achieves a global optimum but is also applicable to discontinuous and non-differentiable functions. Bayesian optimization uses Gaussian process regression for this purpose. Therefore, I will first provide a brief explanation of Gaussian process regression. Next, we will demonstrate an algorithm for finding the maximum points of functions, including examples of functions with one and two variables, discontinuous functions, and optimization problems with constraints. These examples will reveal the effectiveness and limitations of Bayesian optimization.