Tutorial: Optimization via simulation with Bayesian statistics and dynamic programming

Abstract

Bayesian statistics comprises a powerful set of methods for analyzing simulated systems. Combined with dynamic programming and other methods for sequential decision making under uncertainty, Bayesian methods have been used to design algorithms for finding the best of several simulated systems. When the dynamic program can be solved exactly, these algorithms have optimal average-case performance. In other situations, this dynamic programming analysis supports the development of approximate methods with sub-optimal but nevertheless good average-case performance. These methods with good average-case performance are particularly useful when the cost of simulation prevents the use of procedures with worstcase statistical performance guarantees. We provide an overview of Bayesian methods used for selecting the best, providing an in-depth treatment of the simpler case of ranking and selection with independent priors appropriate for smaller-scale problems, and then discussing how these same ideas can be applied to correlated priors appropriate for large-scale problems.

Bayesian Optimization for Materials Design

“…Some work has been done toward the goal of developing such an optimal algorithm [22], but computing the optimal algorithm remains out of reach. Optimal strategies have been computed for other closely related problems in optimization of expensive noisy functions, including stochastic root-finding [51], multiple comparisons with a standard [53], and small instances of discrete noisy optimization with normally distributed noise (also called "ranking and selection") [13].…”

Upper bounds on the Bayes-optimal procedure for ranking & selection with independent normal priors

“…This formulation of the sequential Bayesian R&S problem with independent normal samples, known sampling variance, independent normal prior, infinite horizon, and sampling costs, follows that of Powell 2008, Chick andFrazier 2012), and is quite similar to the model in (Chick and Gans 2009), which assumes a discount factor, and to the model in (Frazier, Powell, and Dayanik 2008), which assumes a finite horizon and no discounting.…”

“…We specifically consider the Bayesian formulation, for which early work dates to Raiffa and Schlaifer (1968), with recent surveys Chick (2006) and Frazier (2012). The other mathematical formulations of the problem are the indifference-zone formulation (see the monograph Bechhofer, Santner, and Goldsman (1995) and the survey Kim and Nelson (2006)); the optimal computing budget allocation, or OCBA (Chen and Lee 2010); and the large-deviations approach (Glynn and Juneja 2004).…”

“…These equations have been used to compute Bayes-optimal procedures for problems with one alternative of unknown value and one of known value Gans 2009, Chick andFrazier 2012), and for problems with two alternatives of unknown value (Frazier, Powell, and Dayanik 2008). However, for problems with more than a few alternatives, solving these dynamic programming equations becomes computationally infeasible, due to the curse of dimensionality (Powell 2007).…”

Asymptotic validity of the Bayes-inspired Indifference Zone procedure: The non-normal known variance case

“…Some compilations of the theory developed in the area can be found in R. E. Bechhofer (1995), Swisher, Jacobson, and Yücesan (2003), Kim and Nelson (2006) and Kim and Nelson (2007). Other approaches, beyond the indifference-zone approach, include the Bayesian approach (Frazier 2012), the optimal computing budget allocation approach (Chen and Lee 2010), the large deviations approach (Glynn and Juneja 2004), and the probability of good selection guarantee (Nelson and Banerjee 2001). The last approach is similar to the indifference-zone formulation, but provides a more robust guarantee.…”