Arnoldi-based Sampling for High-dimensional Optimization using Imperfect Data

Abstract

We present a sampling strategy suitable for optimization problems characterized by high-dimensional design spaces and noisy outputs. Such outputs can arise, for example, in time-averaged objectives that depend on chaotic states. The proposed sampling method is based on a generalization of Arnoldi's method used in Krylov iterative methods. We show that Arnoldi-based sampling can effectively estimate the dominant eigenvalues of the underlying Hessian, even in the presence of inaccurate gradients. This spectral information can be used to build a low-rank approximation of the Hessian in a quadratic model of the objective. We also investigate two variants of the linear term in the quadratic model: one based on step averaging and one based on directional derivatives. The resulting quadratic models are used in a trust-region optimization framework called the Stochastic Arnoldi's Method (SAM). Numerical experiments highlight the potential of SAM relative to conventional derivative-based and derivative-free methods when the design space is high-dimensional and noisy.

Hessian-based Dimension Reduction for Optimization Under Uncertainty

“…At the end of its mth iteration, Arnoldi's method produces an m × m upper Hessenberg matrix, H m . The eigenvalues of the symmetric part of H m provide good estimates for the dominant eigenvalues of ∇ 2 ξ J [44]. The corresponding Ritz-approximate eigenvectors of ∇ 2 ξ J can be obtained by multiplying the eigenvectors of the symmetric part of H m with the orthonormal bases…”

“…(11) is a trial step in the optimization routine. We direct the reader to [44] for more information on the use of Arnoldi's method for optimization.…”