Efficient Natural Gradient Descent Methods for Large-Scale Optimization Problems

Abstract

We propose an efficient numerical method for computing natural gradient descent directions with respect to a generic metric in the state space. Our technique relies on representing the natural gradient direction as a solution to a standard least-squares problem. Hence, instead of calculating, storing, or inverting the information matrix directly, we apply efficient methods from numerical linear algebra to solve this least-squares problem. We treat both scenarios where the derivative of the state variable with respect to the parameter is either explicitly known or implicitly given through constraints. We apply the QR decomposition to solve the least-squares problem in the former case and utilize the adjoint-state method to compute the natural gradient descent direction in the latter case. As a result, we can reliably compute several natural gradient descents, including the Wasserstein natural gradient, for a large-scale parameter space with thousands of dimensions, which was believed to be out of reach. Finally, our numerical results shed light on the qualitative differences among the standard gradient descent method and various natural gradient descent methods based on different metric spaces in large-scale nonconvex optimization problems.

Learning Dynamical Systems From Invariant Measures

“…The adjoint-state method is an efficient technique by which we can evaluate the derivative ∂ θ J , as the computation time is largely independent of the size of θ. One can derive the adjoint-state method for gradient computations by differentiating the discrete constraint [52], which in our case is the eigenvector problem…”

Norm-dependent convergence and stability of the inverse scattering series for diffuse and scalar waves

“…It could also lead to different gradient flow formulations when passing the Fréchet derivative to the gradient, thus giving rise to different gradient descent algorithms for solving such nonconvex optimization problems. Both choices affect the convergence rate and potentially change the stationary points to which the iterative gradient-based algorithm converges, even with the same initial guess [34]. We will demonstrate this later in section 6.…”