A Unified Adaptive Tensor Approximation Scheme to Accelerate Composite Convex Optimization

Jiang, Bo; Lin, Tianyi; Zhang, Shuzhong

doi:10.1137/19m1286025

Cited by 15 publications

(9 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Proof. We follow [20] but we provide the proof details for the reader's convenience. First, we observe that multiplying M k by two will not stop until the line search stopping criterion is satisfied.…”

Section: Properties Of the Apdamd Algorithmmentioning

confidence: 99%

On Efficient Optimal Transport: An Analysis of Greedy and Accelerated Mirror Descent Algorithms

Lin,

Ho,

Jordan

2019

Preprint

Self Cite

View full text Add to dashboard Cite

We provide theoretical analyses for two algorithms that solve the regularized optimal transport (OT) problem between two discrete probability measures with at most n atoms. We show that a greedy variant of the classical Sinkhorn algorithm, known as the Greenkhorn algorithm, can be improved to O n 2 ε 2 , improving on the best known complexity bound of O n 2 ε 3 . Notably, this matches the best known complexity bound for the Sinkhorn algorithm and helps explain why the Greenkhorn algorithm can outperform the Sinkhorn algorithm in practice. Our proof technique, which is based on a primal-dual formulation and a novel upper bound for the dual solution, also leads to a new class of algorithms that we refer to as adaptive primal-dual accelerated mirror descent (APDAMD) algorithms. We prove that the complexity of these algorithms is O n 2 γ 1/2 ε , where γ > 0 refers to the inverse of the strong convexity module of Bregman divergence with respect to • ∞ . This implies that the APDAMD algorithm is faster than the Sinkhorn and Greenkhorn algorithms in terms of ε. Experimental results on synthetic and real datasets demonstrate the favorable performance of the Greenkhorn and APDAMD algorithms in practice.

show abstract

Section: Properties Of the Apdamd Algorithmmentioning

confidence: 99%

On Efficient Optimal Transport: An Analysis of Greedy and Accelerated Mirror Descent Algorithms

Lin,

Ho,

Jordan

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, we also show that (while somewhat helpful for O cr with a conservative choice of H), adding momentum to well-tuned or adaptive second-order methods is harmful in logistic regression: simply iterating our oracle-or, better yet, applying Newton's method-dramatically outperforms all "accelerated" algorithms. This important fact seems to have gone unobserved in the literature on accelerated second-order methods, despite logistic regression appearing in many related experiments [40,16,30,25]. Simply iterating our adaptive oracle outperforms the classical accelerated gradient descent, and performs comparably to L-BFGS.…”

Section: Our Contributionsmentioning

confidence: 94%

Optimal and Adaptive Monteiro-Svaiter Acceleration

Carmon¹,

Hausler²,

Jambulapati³

et al. 2022

Preprint

View full text Add to dashboard Cite

We develop a variant of the Monteiro-Svaiter (MS) acceleration framework that removes the need to solve an expensive implicit equation at every iteration. Consequently, for any p ≥ 2 we improve the complexity of convex optimization with Lipschitz pth derivative by a logarithmic factor, matching a lower bound. We also introduce an MS subproblem solver that requires no knowledge of problem parameters, and implement it as either a second-or first-order method via exact linear system solution or MinRes, respectively. On logistic regression our method outperforms previous second-order acceleration schemes, but under-performs Newton's method; simply iterating our first-order adaptive subproblem solver performs comparably to L-BFGS.∞ regression [8,12], minimizing functions with Hölder continuous higher derivatives [40], and distributionally-robust optimization [13,11].

show abstract

“…We propose a novel way to update the regularization parameters in the third-order models used to compute trial points. Different from existing adaptive tensor methods [13,16], in our new methods the regularization parameters are adjusted taking into account the progress of the inner solver. When the regularization parameter is sufficiently large, the inner solver is guaranteed to have linear rate of convergence.…”

Section: Motivationmentioning

confidence: 99%

Adaptive Third-Order Methods for Composite Convex Optimization

Grapiglia¹,

Nesterov²

2022

Preprint

View full text Add to dashboard Cite

In this paper we propose third-order methods for composite convex optimization problems in which the smooth part is a three-times continuously differentiable function with Lipschitz continuous third-order derivatives. The methods are adaptive in the sense that they do not require the knowledge of the Lipschitz constant. Trial points are computed by the inexact minimization of models that consist in the nonsmooth part of the objective plus a quartic regularization of third-order Taylor polynomial of the smooth part. Specifically, approximate solutions of the auxiliary problems are obtained by using a Bregman gradient method as inner solver. Different from existing adaptive approaches for highorder methods, in our new schemes the regularization parameters are adjusted taking into account the progress of the inner solver. With this technique, we show that the basic method finds an ǫ-approximate minimizer of the objective function performing at most O | log(ǫ)|ǫ − 1 3 iterations of the inner solver. An accelerated adaptive third-order method is also presented with total inner iteration complexity of O | log(ǫ)|ǫ − 1 4 .

show abstract

A Unified Adaptive Tensor Approximation Scheme to Accelerate Composite Convex Optimization

Cited by 15 publications

References 31 publications

On Efficient Optimal Transport: An Analysis of Greedy and Accelerated Mirror Descent Algorithms

On Efficient Optimal Transport: An Analysis of Greedy and Accelerated Mirror Descent Algorithms

Optimal and Adaptive Monteiro-Svaiter Acceleration

Adaptive Third-Order Methods for Composite Convex Optimization

Contact Info

Product

Resources

About