Analysis Methods for Computer Experiments: How to Assess and What Counts?

Chen, Hao; Loeppky, Jason L.; Sacks, Jerome; Welch, William J.

doi:10.1214/15-sts531

Cited by 45 publications

(32 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The LH design is also often used along with other constraints, e.g., the maximin Latin Hypercube maximizes the minimum Euclidean distance in the LH samples. It has been shown that the GaSP emulator based on maximin LH samples has a clear advantage compared to the uniform design in terms of prediction (see, e.g., Chen et al (2016)). For these reasons, we recommend the use of the LH design, rather than the uniform design or lattice designs.…”

Section: Gasp Emulatormentioning

confidence: 99%

RobustGaSP: Robust Gaussian Stochastic Process Emulation in R

Gu¹,

Palomo²,

Berger³

2019

The R Journal

View full text Add to dashboard Cite

Gaussian stochastic process emulation is a powerful tool for approximating computationally intensive computer models. However, estimation of parameters in the GaSP emulator is a challenging task. No closed-form estimator is available and many numerical problems arise with standard estimates, e.g., the maximum likelihood estimator. In this package, we implement a marginal posterior mode estimator, for special priors and parameterizations, an estimation method that meets the robust parameter estimation criteria discussed in ; mathematical reasons are provided therein to explain why robust parameter estimation can greatly improve predictive performance of the emulator. In addition, inert inputs (inputs that almost have no effect on the variability of a function) can be identified from the marginal posterior mode estimation, at no extra computational cost. The package also implements the parallel partial Gaussian stochastic process (PP GaSP) emulator (Gu and Berger (2016)) for the scenario where the computer model has multiple outputs on e.g. spatialtemporal coordinates. The package can be operated in a default mode, but also allows numerous user specifications, such as the capability of specifying trend functions and noise terms. Examples are studied herein to highlight the performance of the package in terms of out-of-sample prediction.

show abstract

Section: Gasp Emulatormentioning

confidence: 99%

RobustGaSP: Robust Gaussian Stochastic Process Emulation in R

Gu¹,

Palomo²,

Berger³

2019

The R Journal

View full text Add to dashboard Cite

show abstract

“…OK uses the constant = E(y), whereas universal Kriging (UK) uses a trend (e.g., E(y) = 0 x); details on UK are found in Kleijnen (2015, pp. 197-198) and also in Chen et al (2016) and Mukhopadhyay et al (2016).…”

Section: Kriging: Basicsmentioning

confidence: 92%

“…206-207). The nugget e¤ect in Kriging metamodels of deterministic simulation models is also discussed in Chen et al (2016), Mukhopadhyay et al (2016), and Peng and Wu (2014).…”

Section: Conclusion and Future Researchmentioning

confidence: 99%

See 1 more Smart Citation

Prediction for Big Data Through Kriging: Small Sequential and One-Shot Designs

Kleijnen

Beers

2018

SSRN Journal

View full text Add to dashboard Cite

Kriging or Gaussian process (GP) modeling is an interpolation method that assumes the outputs (responses) are more correlated, the closer the inputs (explanatory or independent variables) are. A GP has unknown (hyper)parameters that must be estimated; the standard estimation method uses the "maximum likelihood" criterion. However, big data make it hard to compute the estimates of these GP parameters, and the resulting Kriging predictor and the variance of this predictor. To solve this problem, some authors select a relatively small subset from the big set of previously observed "old" data; their method is sequential and depends on the variance of the Kriging predictor. The resulting designs turn out to be "local"; i.e., most design points are concentrated around the point to be predicted. We develop three alternative one-shot methods that do not depend on GP parameters: (i) select a small subset such that this subset still covers the original input space-albeit coarser; (ii) select a subset with relatively many-but not all-combinations close to the new combination that is to be predicted, and (iii) select a subset with the nearest neighbors (NNs) of this new combination. To evaluate these designs, we compare their squared prediction errors in several numerical (Monte Carlo) experiments. These experiments show that our NN design is a viable alternative for the more sophisticated sequential designs.

show abstract

“…There are many variants on emulation, with some practitioners preferring no regressors (Chen et al, 2016), different types of correlation function (including no correlation) (Kaufman et al, 2011;Salter and Williamson, 2016), and different priors, π(β, φ), with some leading to partially analytic posterior inference (Haylock and O'Hagan, 1996). As history matching only requires posterior means and variances of the emulator, Bayes linear analogues are sometimes used (Vernon et al, 2010).…”

Section: Calibration Methodologies and The Terminal Casementioning

confidence: 99%

Uncertainty Quantification for Computer Models With Spatial Output Using Calibration-Optimal Bases

Salter

Williamson

Scinocca³

et al. 2019

Journal of the American Statistical Association

View full text Add to dashboard Cite

The calibration of complex computer codes using uncertainty quantification (UQ) methods is a rich area of statistical methodological development. When applying these techniques to simulators with spatial output, it is now standard to use principal component decomposition to reduce the dimensions of the outputs in order to allow Gaussian process emulators to predict the output for calibration. We introduce the 'terminal case', in which the model cannot reproduce observations to within model discrepancy, and for which standard calibration methods in UQ fail to give sensible results. We show that even when there is no such issue with the model, the standard decomposition on the outputs can and usually does lead to a terminal case analysis. We present a simple test to allow a practitioner to establish whether their experiment will result in a terminal case analysis, and a methodology for defining calibration-optimal bases that avoid this whenever it is not inevitable. We present the optimal rotation algorithm for doing this, and demonstrate its efficacy for an idealised example for which the usual principal component methods fail. We apply these ideas to the CanAM4 model to demonstrate the terminal case issue arising for climate models. We discuss climate model tuning and the estimation of model discrepancy within this context, and show how the optimal rotation algorithm can be used in developing practical climate model tuning tools. * The authors gratefully acknowledge support from EPSRC fellowship No. EP/K019112/1 and support from the NSERC funded Canadian Network for Regional Climate and Weather Processes (CNRCWP). We would also like to thank Yanjun Jiao for managing our ensembles of CanAM4.

show abstract

Analysis Methods for Computer Experiments: How to Assess and What Counts?

Cited by 45 publications

References 38 publications

RobustGaSP: Robust Gaussian Stochastic Process Emulation in R

RobustGaSP: Robust Gaussian Stochastic Process Emulation in R

Prediction for Big Data Through Kriging: Small Sequential and One-Shot Designs

Uncertainty Quantification for Computer Models With Spatial Output Using Calibration-Optimal Bases

Contact Info

Product

Resources

About