Learning Arbitrary Quantities of Interest from Expensive Black-Box Functions through Bayesian Sequential Optimal Design

Abstract

Estimating arbitrary quantities of interest (QoIs) that are non-linear operators of complex, expensive-to-evaluate, black-box functions is a challenging problem due to missing domain knowledge and finite information acquisition budgets. Bayesian optimal design of experiments (BODE) is a family of methods that identify an optimal design of experiments (DOE) under different contexts, such as learning a response surface, estimating a statistical expectation, solving an optimization problem, etc., using only in a limited number of function evaluations. Under BODE methods, sequential design of experiments (SDOE) accomplishes this task by selecting an optimal sequence of experiments. SDOE methods use data-driven probabilistic surrogate models for emulating the expensive black-box function. Probabilistic predictions from the surrogate model are used to define an information acquisition function (IAF) which quantifies the marginal value contributed or the expected information gained by a hypothetical experiment. The next experiment is selected by maximizing the IAF. A generally applicable IAF is the expected information gain (EIG) about a QoI as captured by the expectation of the Kullback-Leibler divergence between the predictive distribution of the QoI after doing a hypothetical experiment and the current predictive distribution about the same QoI. Despite its intuitive appeal, applications of EIG have been limited due to the difficulty associated with estimating it.To overcome this barrier, in this work, we develop a practical numerical estimator of the EIG for arbitrary QoIs. We model the underlying information source as a fully-Bayesian, non-stationary Gaussian process (FB-NSGP), derive an approximation of the information gain of a hypothetical experiment about an arbitrary QoI conditional on the hyper-parameters, and estimate the EIG about the same QoI using sampling averages to integrate over the posterior of the hyper-parameters and the potential experimental outcomes. We demonstrate the performance of our method in four numerical examples and a practical engineering problem of steel wire manufacturing. The method is compared to two classic SDOE methods: random sampling and uncertainty sampling. The results show a mixed outcome where the proposed methods converges sooner with increasing