Comparison of Kriging-Based Methods for Simulation Optimization with Heterogeneous Noise

Abstract

a b s t r a c tIn this article we investigate the unconstrained optimization (minimization) of the performance of a system that is modeled through a discrete-event simulation. In recent years, several algorithms have been proposed which extend the traditional Kriging-based simulation optimization algorithms (assuming deterministic outputs) to problems with noise. Our objective in this paper is to compare the relative performance of a number of these algorithms on a set of well-known analytical test functions, assuming different patterns of heterogeneous noise. We also apply the algorithms to a popular inventory test problem. The conclusions and insights obtained may serve as a useful guideline for researchers aiming to apply Kriging-based algorithms to solve engineering and/or business problems, and may be useful in the development of future algorithms.

Evaluating Gaussian Process Metamodels and Sequential Designs for Noisy Level Set Estimation

“…Our analysis is driven by the primal effect of noise on contour-finding algorithms. This effect was already documented in related studies, such as that of Jalali et al [19] who observed the strong impact of (•) on performance of Bayesian optimization. Consequently, specialized metamodeling frameworks and acquisition functions are needed that can best handle the stochasticity for the given loss specification.…”

“…The main goal of this article is to present a comprehensive assessment of GP-based surrogates for stochastic contour-finding. In that sense, our analysis complements the work of Picheny et al [31] and Jalali et al [19], who benchmarked GP metamodels for Bayesian optimization where the objective is to evaluate max x f (x).…”

“…For the response modeling aspect, GP regression, or kriging, has emerged as the most popular nonparametric approach for both deterministic and stochastic black-box functions [4,16,29,19]. GPs have also been widely used for the level-set estimation problem; see [9,15,18,30] and [32].…”

Analyzing Stochastic Computer Models: A Review with Opportunities

“…An R package for implementing several of these choices is available in DiceOptim (Picheny et al (2016); Picheny and Ginsbourger (2014)). Jalali et al (2017) also do a similar comparison for heteroscedastic noise.…”

Analyzing Stochastic Computer Models: A Review with Opportunities

“…Alternative criteria to EI, specifically designed for stochastic problems, also exist within the same framework. Jalali et al (2016) provide a brief summary of some alternative criteria, as well as an in depth comparison between them. An R package for implementing several stochastic optimization methods (including some choices for the replacements of y max ), using the homGP model from Section 3.1, is available in (DiceOptim; Picheny et al (2016); Picheny and Ginsbourger (2014)).…”