A statistical method for global optimization

Abstract

An algorithm for finding global optima using statistical prediction is presented. Assuming a random function model, lower confidence bounds on predicted values are used for sequential selection of evaluation points and as a convergence criterion. Performance comparision with published results on several test functions indicate that the procedure is very efficient in finding the global optimum of a multimodal function, and in terminating with relatively few evaluations. I . INTRODUCTIONcurrent minimum is not unlikely. There are two possibilities for this -either there is not much information about the objective in a region, or the information we have indicates that there is likely to be a lower value of the objective. Thus, SDO provides an "intelligent" way for the search to keep track of promising regions and not to ignore unexplored regions.The results reported here indicate SDO is highly efficient vis a vis the number of function evaluations when compared to other methods. Some features include:It does not require tshe calculat,ion of derivatives;The explicit form of the objective function need not he known; The problem of optimizing a function possibly having many local extrema arises frequently in practice.A thorough search in the domain of interest is almost always impractical since it requires too many function evaluations. Therefore, one has to rely on a limited search, either random or systematic, to locate the global extremum. Here, we propose a search method based onThe method easily extends to the situation in which only noisy evaluations of the object,ive function are available;There is a convergence criterion which Provides a natural stopping rule;The statistical computations involve linear algebra operations which are readily vectorized and easily adapted to parallel architectures. a random function model for the objective function. We refer to the method as Sequential Design for Optimization (SDO). We briefly describe SDO. The random function model allows us to construct an approximation to t,he objective function known as the Best Linear Unbiased Predictor (BLUP). It also allows us to give a statistical bound on the approximation error. Suppose for convenience that our goal is minimization. Then using the BLUP and error bound, we can determine if a point in the domain of interest is unlikely to have an objective function value below a given value, namely the currently observed minimum value of t,he objective. The search is restricted to regions of the domain where a smaller value than the Manuscript received August 1, 1992.However, there is still some work needed as the current version is restricted to a fixed grid of values of t.he argument. To make SDO feasible in high dimensions, it will be necessary to develop a more adaptive procedure for selecting a set of candidate values for the input variable. Nonetheless, the results present,ed here are very promising. Section I1 introduces the model, Best Linear Unbiased Predictors (BLUP's), and a description of SDO. Section 111 presents our results and...

Weighted Gradient-Enhanced Kriging for High-Dimensional Surrogate Modeling and Design Optimization

“…A variety of surrogate models, such as PRSM, kriging and its variants (GEK [47], cokriging [33], HK [34]), RBFs, ANN, SVR [22], etc., were implemented. A couple of infill-sampling criteria and dedicated constraint handling methods were implemented, such as minimizing surrogate predictor (MSP) [76], expected improvement [77,78], probability of improvement [5], mean-squared error (MSE) [79,80], lower-confidence bounding [51,81], target searching [74], and parallel infilling [30]. Some well-accepted and highly matured optimization algorithms, such as Hooke and Jeeves pattern search, Simplex, BFGS quasi-Newton's method, sequential quadratic programming (SQP), and single/multi objective genetic algorithms (GAs) [82], are employed to solve the suboptimization(s), in which the cost function(s) and constraint function(s) are evaluated by the cheap surrogate models.…”

New horizons in sphere-packing theory, part II: lattice-based derivative-free optimization via global surrogates

“…We distinguish between two scenarios in which the polling algorithm as described thus far must be adjusted to avoid violating the (n − 1)-dimensional boundaries 5 of the feasible domain. In the first scenario, the CMP is relatively far (that is, greater than ρ n but less than 2ρ n ) from the boundary of the feasible domain, and thus one or more of the poll points as determined by one of the algorithms proposed in §II-B might land slightly outside this boundary.…”

“…To achieve this, consider the minimization of J(x) =f (x) − c · s 2 (x), where c is some constant (see Cox &John 1997 andJones 2001). A search coordinated by this function will tend to explore regions of parameter space where both the predictor of the function value is relatively low and the uncertainty of this prediction in the Kriging model is relatively high.…”

The Knowledge-Gradient Algorithm for Sequencing Experiments in Drug Discovery

“…In addition, there is the interval estimation (IE) policy (Kaelbling 1993), which combines exploration with exploitation by measuring the compound for which a particular linear combination of the posterior mean and posterior standard deviation is largest. IE was applied to BGO in Cox and John (1997), where it was called SDO, or sequential design for optimization.…”