Second-order Conditional Gradient Sliding

Carderera, Alejandro; Pokutta, Sebastian

doi:10.48550/arxiv.2002.08907

Cited by 6 publications

(8 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Usually, complexity of this step can be estimated as O(n 3 ) arithmetic operations, which comes from the cost of computing a suitable factorization for the Hessian matrix. Alternatively, Hessian-free gradient methods can be applied, for computing an inexact step (see [5,4]).…”

Section: Contracting-domain Newton Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Convex optimization based on global lower second-order models

Doikov,

Nesterov

2020

Preprint

View full text Add to dashboard Cite

In this paper, we present new second-order algorithms for composite convex optimization, called Contracting-domain Newton methods. These algorithms are affine-invariant and based on global second-order lower approximation for the smooth component of the objective. Our approach has an interpretation both as a second-order generalization of the conditional gradient method, or as a variant of trust-region scheme. Under the assumption, that the problem domain is bounded, we prove O(1/k 2 ) global rate of convergence in functional residual, where k is the iteration counter, minimizing convex functions with Lipschitz continuous Hessian. This significantly improves the previously known bound O(1/k) for this type of algorithms. Additionally, we propose a stochastic extension of our method, and present computational results for solving empirical risk minimization problem.

show abstract

Section: Contracting-domain Newton Methodsmentioning

confidence: 99%

“…are determined by the dataset 4 . First, we compare the performance of Contracting-Domain Newton Method (Algorithm 1) and Aggregating Newton Method (Algorithm 3) with first-order optimization schemes: Frank-Wolfe algorithm [14], the classical Gradient Method, and the Fast Gradient Method [26].…”

Section: Stochastic Finite-sum Minimizationmentioning

confidence: 99%

Convex optimization based on global lower second-order models

Doikov,

Nesterov

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…We used the fact that ∇h(x * ), z (t) − x * ≥ 0 by first order optimality and the fact that z (t+1) − x * ≤ z (t) − x * (see (42)) in (44). The inequality in (45) follows by the smoothness of h. Further, we used Young's inequality † † † in (47). Finally, we used the PL-inequality (23) (which states that w(z (t+1) ) ≤ ∇h(z (t+1) ) 2 /2µ) and the fact that c ≥ ∇h(x * ) 2 in (48).…”

Section: Results For the A 2 Fw Algorithmmentioning

confidence: 99%

“…or prioritizing in-face steps [45], or theoretical results such as [46] and [47] show that FW variants must use active sets that containing the optimal solution after crossing a polytope dependent radius of convergence. These results, however, do not use combinatorial properties of previous minimizers or detect tight sets with provable guarantees and round to those.…”

Section: Logistic Lossmentioning

confidence: 99%

Reusing Combinatorial Structure: Faster Iterative Projections over Submodular Base Polytopes

Moondra¹,

Mortagy²,

Gupta³

2021

Preprint

View full text Add to dashboard Cite

Optimization algorithms such as projected Newton's method, FISTA, mirror descent and its variants enjoy near-optimal regret bounds and convergence rates, but suffer from a computational bottleneck of computing "projections" in potentially each iteration (e.g., O(T 1/2 ) regret of online mirror descent) [1,2,3,4]. On the other hand, conditional gradient variants solve a linear optimization in each iteration, but result in suboptimal rates (e.g., O(T 3/4 ) regret of online Frank-Wolfe) [5,6,7,8]. Motivated by this trade-off in runtime v/s convergence rates, we consider iterative projections of close-by points over widely-prevalent submodular base polytopes B(f ). We develop a toolkit to speed up the computation of projections using both discrete and continuous perspectives (e.g., [9,10,11]). We subsequently adapt the away-step Frank-Wolfe algorithm to use this information and enable early termination. For the special case of cardinality based submodular polytopes, we improve the runtime of computing certain Bregman projections by a factor of Ω(n/ log(n)). Our theoretical results show orders of magnitude reduction in runtime in preliminary computational experiments.

show abstract

“…For this reason, FW methods fall into the category of projectionfree methods [64]. Furthermore, the method can be used to approximately solve quadratic subproblems in accelerated schemes, an approach usually referred to as conditional gradient sliding (see, e.g., [20,65]).…”

Section: Introductionmentioning

confidence: 99%

Frank-Wolfe and friends: a journey into projection-free first-order optimization methods

Bomze¹,

Rinaldi²,

Zeffiro³

2021

Preprint

View full text Add to dashboard Cite

Invented some 65 years ago in a seminal paper by Marguerite Straus-Frank and Philip Wolfe, the Frank-Wolfe method recently enjoys a remarkable revival, fuelled by the need of fast and reliable first-order optimization methods in Data Science and other relevant application areas. This review tries to explain the success of this approach by illustrating versatility and applicability in a wide range of contexts, combined with an account on recent progress in variants, both improving on the speed and efficiency of this surprisingly simple principle of firstorder optimization.

show abstract

Second-order Conditional Gradient Sliding

Cited by 6 publications

References 22 publications

Convex optimization based on global lower second-order models

Convex optimization based on global lower second-order models

Reusing Combinatorial Structure: Faster Iterative Projections over Submodular Base Polytopes

Frank-Wolfe and friends: a journey into projection-free first-order optimization methods

Contact Info

Product

Resources

About