Computational Enhancements to Bayesian Design of Experiments Using Gaussian Processes

Weaver, Brian P.; Williams, Brian J.; Anderson‐Cook, Christine M.; Higdon, David

doi:10.1214/15-ba945

Cited by 37 publications

(24 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…That is: how to allocate a set of unique locations, and the degree of replication thereon, to obtain the best overall fit to the data. That sentiment has been echoed independently in several recent publications (Kleijnen, 2015;Weaver et al, 2016;Jalali et al, 2017;Horn et al, 2017). The standard approach of allocating a uniform number of replicates leaves plenty of room for improvement.…”

Section: Introductionmentioning

confidence: 99%

Replication or Exploration? Sequential Design for Stochastic Simulation Experiments

et al. 2018

View full text Add to dashboard Cite

We investigate the merits of replication, and provide methods for optimal design (including replicates), with the goal of obtaining globally accurate emulation of noisy computer simulation experiments. We first show that replication can be beneficial from both design and computational perspectives, in the context of Gaussian process surrogate modeling. We then develop a lookahead based sequential design scheme that can determine if a new run should be at an existing input location (i.e., replicate) or at a new one (explore). When paired with a newly developed heteroskedastic Gaussian process model, our dynamic design scheme facilitates learning of signal and noise relationships which can vary throughout the input space. We show that it does so efficiently, on both computational and statistical grounds. In addition to illustrative synthetic examples, we demonstrate performance on two challenging real-data simulation experiments, from inventory management and epidemiology.

show abstract

Section: Introductionmentioning

confidence: 99%

Replication or Exploration? Sequential Design for Stochastic Simulation Experiments

et al. 2018

View full text Add to dashboard Cite

show abstract

“…A state-of-the-art optimization algorithm for such scenarios is Bayesian optimization [36]. Bayesian optimization has already been effectively used in experimental design problems [22,37], and it was therefore our optimizer of choice. Bayesian optimization is a global optimization algorithm which uses a surrogate probabilistic model of the utility function to decide which values of optimized variables to evaluate next; this allows it to efficiently optimize over expensive utility functions, at the cost of continuously updating the surrogate model (see S1 Appendix for technical details).…”

Section: Optimizing Associative Learning Experimentsmentioning

confidence: 99%

Computational optimization of associative learning experiments

Melinščak

Bach

2020

PLoS Comput Biol

View full text Add to dashboard Cite

With computational biology striving to provide more accurate theoretical accounts of biological systems, use of increasingly complex computational models seems inevitable. However, this trend engenders a challenge of optimal experimental design: due to the flexibility of complex models, it is difficult to intuitively design experiments that will efficiently expose differences between candidate models or allow accurate estimation of their parameters. This challenge is well exemplified in associative learning research. Associative learning theory has a rich tradition of computational modeling, resulting in a growing space of increasingly complex models, which in turn renders manual design of informative experiments difficult. Here we propose a novel method for computational optimization of associative learning experiments. We first formalize associative learning experiments using a low number of tunable design variables, to make optimization tractable. Next, we combine simulation-based Bayesian experimental design with Bayesian optimization to arrive at a flexible method of tuning design variables. Finally, we validate the proposed method through extensive simulations covering both the objectives of accurate parameter estimation and model selection. The validation results show that computationally optimized experimental designs have the potential to substantially improve upon manual designs drawn from the literature, even when prior information guiding the optimization is scarce. Computational optimization of experiments may help address recent concerns over reproducibility by increasing the expected utility of studies, and it may even incentivize practices such as study pre-registration, since optimization requires a pre-specified analysis plan. Moreover, design optimization has the potential not only to improve basic research in domains such as associative learning, but also to play an important role in translational research. For example, design of behavioral and physiological diagnostic tests in the nascent field of computational psychiatry could benefit from an optimization-based approach, similar to the one presented here.

show abstract

“…In linear cases, the OED problem includes various criteria such as the D-optimal design criterion and the A-optimal design criterion. The D-optimal design criterion seeks to maximize the determinant of the information matrix of the design, whereas the A-optimal design criterion considers minimizing the trace of the inverse of the information matrix which results in minimizing the average variance [7][8][9].…”

Section: Introductionmentioning

confidence: 99%

Gaussian Process Based Expected Information Gain Computation for Bayesian Optimal Design

Liao

2020

Entropy

View full text Add to dashboard Cite

Optimal experimental design (OED) is of great significance in efficient Bayesian inversion. A popular choice of OED methods is based on maximizing the expected information gain (EIG), where expensive likelihood functions are typically involved. To reduce the computational cost, in this work, a novel double-loop Bayesian Monte Carlo (DLBMC) method is developed to efficiently compute the EIG, and a Bayesian optimization (BO) strategy is proposed to obtain its maximizer only using a small number of samples. For Bayesian Monte Carlo posed on uniform and normal distributions, our analysis provides explicit expressions for the mean estimates and the bounds of their variances. The accuracy and the efficiency of our DLBMC and BO based optimal design are validated and demonstrated with numerical experiments.

show abstract

Computational Enhancements to Bayesian Design of Experiments Using Gaussian Processes

Cited by 37 publications

References 26 publications

Replication or Exploration? Sequential Design for Stochastic Simulation Experiments

Replication or Exploration? Sequential Design for Stochastic Simulation Experiments

Computational optimization of associative learning experiments

Gaussian Process Based Expected Information Gain Computation for Bayesian Optimal Design

Contact Info

Product

Resources

About