A New Perspective of Accelerated Gradient Methods: The Controlled Invariant Manifold Approach

Abstract

<p>Gradient Descent (GD) is a ubiquitous algorithm for finding the optimal solution to an optimization problem. For reduced computational complexity, the optimal solution <em>x*</em> of the optimization problem must be attained in a minimum number of iterations. For this objective, the paper proposes a genesis of an accelerated gradient algorithm through the controlled dynamical system perspective. The objective of optimally reaching the optimal solution <em>x*</em> where the gradient of ∇<em>f(x*)</em> is zero with a given initial condition <em>x(0)</em> is achieved through control.</p>