Second-Order Corrections for Surrogate-Based Optimization with Model Hierarchies

Abstract

Surrogate-based optimization methods have become established as effective techniques for engineering design problems through their ability to tame nonsmoothness and reduce computational expense. In recent years, supporting mathematical theory has been developed to provide the foundation of provable convergence for these methods. One of the requirements of this provable convergence theory involves consistency between the surrogate model and the underlying truth model that it approximates. This consistency can be enforced through a variety of correction approaches, and is particularly essential in the case of surrogate-based optimization with model hierarchies. First-order additive and multiplicative corrections currently exist which satisfy consistency in values and gradients between the truth and surrogate models at a single point. This paper demonstrates that first-order consistency can be insufficient to achieve acceptable convergence rates in practice and presents new second-order additive, multiplicative, and combined corrections which can significantly accelerate convergence. These second-order corrections may enforce consistency with either the actual truth model Hessian or its finite difference, quasi-Newton, or Gauss-Newton approximation.

A Survey of Projection-Based Model Reduction Methods for Parametric Dynamical Systems

“…Data-fit models include response surface methods that use interpolation or regression of simulation data to fit a model for the system output as a function of the parameters. In the statistical literature, Gaussian processes have been used extensively as data-fit surrogates for complex computational models [140], while data-fit surrogate approaches in optimization include polynomial response surfaces [85,108,137,216], radial basis functions [224], and Kriging models [204]. Stochastic spectral approximations, commonly used in uncertainty quantification, are another form of response surface model.…”

Bayesian methods in engineering design problems.

“…In these cases, we would like to develop an approach to response surface modeling which allows us to construct a response surface based on some low fidelity function evaluations and update the coefficients governing that response surface with a few high fidelity function evaluations. This approach of correcting a lowfidelity response surface and updating it is used in some trust region [Eldred et al, 2004]. A variation on this approach has been developed by Kennedy and O'Hagan (2000), who propose constructing an autoregressive model where a higher-fidelity code output is assumed to be an autoregressive function of the lower fidelity code output.…”

Surrogate-based infill optimization applied to electromagnetic problems

“…One may also use higher order scaling strategies, e.g., where the derivatives of the low-fidelity model are also modified to agree with the high-fidelity model. A more complex combination of both approaches is also possible; for instance Eldred et al [17] propose to write the low-fidelity model as a weighted combination of additive and multiplicative scaling factors. Alternatively, space mapping methods can be utilized.…”