On the von Neumann and Frank--Wolfe Algorithms with Away Steps

Abstract

The von Neumann algorithm is a simple coordinate-descent algorithm to determine whether the origin belongs to a polytope generated by a finite set of points. When the origin is in the interior of the polytope, the algorithm generates a sequence of points in the polytope that converges linearly to zero. The algorithm's rate of convergence depends on the radius of the largest ball around the origin contained in the polytope.We show that under the weaker condition that the origin is in the polytope, possibly on its boundary, a variant of the von Neumann algorithm that includes away steps generates a sequence of points in the polytope that converges linearly to zero. The new algorithm's rate of convergence depends on a certain geometric parameter of the polytope that extends the above radius but is always positive. Our linear convergence result and geometric insights also extend to a variant of the Frank-Wolfe algorithm with away steps for minimizing a convex quadratic function over a polytope.

Frank-Wolfe with a Nearest Extreme Point Oracle

“…Now note that since Algorithm 1 is a decent method (i.e., the function value never increases from one iteration to the next), and so all iterates as well as the optimal set X * are contained within the initial level set L, and thus for any t ≥ 1, we can bound dist(x t , X * ) by D L . Thus, using (13) we have that,…”

“…It is well-known that this rate does not improve even if the objective function is strongly convex (see for instance [4]), a property that is well known to allow for faster convergence rates, and in particular linear rates, for projected/proximal gradient methods [5,6]. Indeed, in recent years there is a significant research effort to design Frank-Wolfe variants with linear convergence rates under strong convexity or the weaker assumption of quadratic growth (see Definition 1 in the sequel), with most efforts focused on the case in which the feasible set is a convex and compact polytope [7,8,9,10,11,12,13,14,15,16].…”

Frank-Wolfe Methods with an Unbounded Feasible Region and Applications to Structured Learning

“…Later on, Guelat and Marcotte [24] show that a modified version of the away-step Frank-Wolfe has a linear convergence guarantee when the objective is strongly convex and the feasible region is a polytope. The linear convergence guarantee of the away-step Frank-Wolfe method has been improved recently with simpler algorithms and a more straightforward analysis, see for example, [31,6,44,43].…”

“…The iterate solution can land on a certain face of the constraint, which avoids this zigzagging phenomenon. Moreover, with away-steps, the Frank-Wolfe method enjoys a linear convergence rate when the objective is further strongly convex and the constraint set is a polytope [31,44,43]. In addition to faster convergence, the away-step Frank-Wolfe method can usually lead to a sparser solution, which is helpful in many applications when sparsity and interpretability are desirable properties of the solution [22].…”

Revisiting Frank-Wolfe for Polytopes: Strict Complementarity and Sparsity

“…In recent years Garber and Hazan [10,12] and then Simon Lacoste Julien and Jaggi [20] presented variants of the Frank-Wolfe method that utilize away steps alongside new analyses, which resulted in provable and explicit linear rates without requiring strict complementarity conditions and without dependence on the location of the optimal solution. These results have encouraged much followup theoretical and empirical work e.g., [2,24,23,14,25,13,26,16,5,15,1,4,21,7], to name a few. However, the linear convergence rates in [10,12,20] and follow-up works depend explicitly on the dimension of the problem (at least linear dependence, i.e., the convergence rate is of the form exp(−Θ(t/d)), where d is the dimension) 1 .…”