Efficient first-order methods for convex minimization: a constructive approach

Abstract

We describe a novel constructive technique for devising efficient first-order methods for a wide range of large-scale convex minimization settings, including smooth, non-smooth, and strongly convex minimization. The technique builds upon a certain variant of the conjugate gradient method to construct a family of methods such that a) all methods in the family share the same worst-case guarantee as the base conjugate gradient method, and b) the family includes a fixed-step first-order method. We demonstrate the effectiveness of the approach by deriving optimal methods for the smooth and non-smooth cases, including new methods that forego knowledge of the problem parameters at the cost of a one-dimensional line search per iteration, and a universal method for the union of these classes that requires a three-dimensional search per iteration. In the strongly convex case, we show how numerical tools can be used to perform the construction, and show that the resulting method offers an improved worst-case bound compared to Nesterov's celebrated fast gradient method. IntroductionConvex optimization plays a central role in many fields of applications, including optimal control, machine learning and signal processing. In particular, when a large number of variables are involved within a convex optimization problem, the use of first-order methods is more and more widespread due to their typically very attractive low computational cost per iteration. This low computational cost comes, however, at a price: first-order methods often suffer from potentially slow convergence speeds, making them appropriate mostly for obtaining low to medium accuracy solutions. Nevertheless, first-order methods remain the methods of choice in many applications and currently receive a lot of attention from the optimization community, which constantly aims at improving them.An effective and fruitful approach used for analyzing and comparing first-order methods is the study of their worst-case behavior through the black-box model. In this setting, methods are only allowed to gain information on the objective through an oracle, which provides the value and the gradient of the objective at selected points.

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

“…However, convergence rates are only one side of the coin; in several applications, e.g., in a data-based setting, also robustness with respect to various kinds of disturbances is a key issue and similar analysis tools have been developed [2,26]. In addition, the problem of designing algorithms specifically tailored to classes of structured optimization problems has only been touched upon so far [8,11,18]. However, in situations where a class of optimization problems needs to be solved repeatedly online, as for example in model predictive control or reinforcement learning, well-performing algorithms are key.…”

Primal–dual accelerated gradient methods with small-dimensional relaxation oracle

“…Universal gradient methods for convex problems were proposed in [29] and extended in [12] for non-convex problems and in [31] for primal-dual setting. Finally, there were some attempts to combine universality with small-dimension relaxation [8,14]. Concerning conjugate gradient methods, we refer the reader to a good survey [2] and the classical book [30].…”

“…Ak+1 , inequality(8) holds. For option b) in the step 4, (8) holds by the choice of a k+1 from the equation…”

A universal modification of the linear coupling method

“…By combining the core idea of Nesterov's Universal Fast Gradient Method with the framework described by Allen-Zhu et al in [1], such a method was devised. As far as it is known to the authors of this paper, our work contains the first example of such a method, although a method utilising exact line search for solving minimization problems with convex Lipschitz continuous objectives was recently constructed by Drori et al [4]. Their work also contains an example of a universal method which uses an exact three dimension subspace minimization on each iteration.…”