Quantile Estimation with Latin Hypercube Sampling

INFORMS Journal on Computing

Beers

2022

Kriging or Gaussian process models are popular metamodels (surrogate models or emulators) of simulation models; these metamodels give predictors for input combinations that are not simulated. To validate these metamodels for computationally expensive simulation models, the analysts often apply computationally efficient cross-validation. In this paper, we derive new statistical tests for so-called leave-one-out cross-validation. Graphically, we present these tests as scatterplots augmented with confidence intervals that use the estimated variances of the Kriging predictors. To estimate the true variances of these predictors, we might use bootstrapping. Like other statistical tests, our tests—with or without bootstrapping—have type I and type II error probabilities; to estimate these probabilities, we use Monte Carlo experiments. We also use such experiments to investigate statistical convergence. To illustrate the application of our tests, we use (i) an example with two inputs and (ii) the popular borehole example with eight inputs. Summary of Contribution: Simulation models are very popular in operations research (OR) and are also known as computer simulations or computer experiments. A popular topic is design and analysis of computer experiments. This paper focuses on Kriging methods and cross-validation methods applied to simulation models; these methods and models are often applied in OR. More specifically, the paper provides the following; (1) the basic variant of a new statistical test for leave-one–out cross-validation; (2) a bootstrap method for the estimation of the true variance of the Kriging predictor; and (3) Monte Carlo experiments for the evaluation of the consistency of the Kriging predictor, the convergence of the Studentized prediction error to the standard normal variable, and the convergence of the expected experimentwise type I error rate to the prespecified nominal value. The new statistical test is illustrated through examples, including the popular borehole model.

Section: Latin Hypercube Designs For Krigingmentioning

confidence: 99%

Statistical Tests for Cross-Validation of Kriging Models

INFORMS Journal on Computing

Beers

2022

“…LHS is also available in the MATLAB Statistics toolbox, the R package, the Open TURNS software, and Sandia's DAKOTA software. Various LHS algorithms are referenced in Kleijnen (2015, p. 200); recent algorithms are detailed in Dong and Nakayama (2017), and Le Guiban et al 2018. Panagiotopoulos et al (2018) uses LHS variants for RBFs-instead of Kriging-metamodels for optimization through genetic algorithms.…”

Section: Latin Hypercube Designs For Krigingmentioning

confidence: 99%

Statistical Tests for Cross-Validation of Kriging Models

Beers²

2019

SSRN Journal

We derive new statistical tests for leave-one-out cross-validation of Kriging models. Graphically, we present these tests as scatterplots augmented with con…dence intervals. We may wish to avoid extrapolation, which we de…ne as prediction of the output for a point that is a vertex of the convex hull of the given input combinations. Moreover, we may use bootstrapping to estimate the true variance of the Kriging predictor. The resulting tests (with or without extrapolation or bootstrapping) have type-I and type-II error probabilities, which we estimate through Monte Carlo experiments. To illustrate the application of our tests, we use an example with two inputs and the popular borehole example with eight inputs.

“…Our algorithm 1 is a (pseudo)algorithm for LHS for option (i) (using midpoints) with n combinations of k inputs, which gives the n k design matrix X L (the subscript L stands for LHS) (various algorithms for LHS are referenced in Kleijnen (2015, p. 200); recent algorithms are detailed in Dong and Nakayama (2017), and Le Guiban et al (2017)):…”

Section: Lhs With Uniform Input Distributionsmentioning

confidence: 99%

Prediction for Big Data Through Kriging: Small Sequential and One-Shot Designs

Beers

2018

SSRN Journal

Kriging or Gaussian process (GP) modeling is an interpolation method that assumes the outputs (responses) are more correlated, the closer the inputs (explanatory or independent variables) are. A GP has unknown (hyper)parameters that must be estimated; the standard estimation method uses the "maximum likelihood" criterion. However, big data make it hard to compute the estimates of these GP parameters, and the resulting Kriging predictor and the variance of this predictor. To solve this problem, some authors select a relatively small subset from the big set of previously observed "old" data; their method is sequential and depends on the variance of the Kriging predictor. The resulting designs turn out to be "local"; i.e., most design points are concentrated around the point to be predicted. We develop three alternative one-shot methods that do not depend on GP parameters: (i) select a small subset such that this subset still covers the original input space-albeit coarser; (ii) select a subset with relatively many-but not all-combinations close to the new combination that is to be predicted, and (iii) select a subset with the nearest neighbors (NNs) of this new combination. To evaluate these designs, we compare their squared prediction errors in several numerical (Monte Carlo) experiments. These experiments show that our NN design is a viable alternative for the more sophisticated sequential designs.