Exact Worst-Case Performance of First-Order Methods for Composite Convex Optimization

Taylor, Adrien; Hendrickx, Julien M.; Glineur, François

doi:10.1137/16m108104x

Cited by 88 publications

(154 citation statements)

References 33 publications

Supporting

Mentioning

153

Contrasting

Order By: Relevance

“…This work takes place within the current effort for the development of systematic/computer-guided analyses and design of optimization algorithms. Among them, a systematic approach to lower bounds (which focuses on quadratic cases) is presented in by Arjevani et al in [1], a systematic use of control theory (via integral quadratic constraints) for developing upper bounds is presented by Lessard et al in [26], and the performance estimation approach, which aims at finding worst-case bounds was originally developed in [13] (see also surveys in [11] and [47]). Those methodologies are mostly presented as tools for performing worst-cases analyses (see the numerous examples in [11,19,47,48,50,51]), however, such techniques were also recently used to develop new methods with improved worst-case complexities.…”

Section: Links With Systematic and Computer-assisted Approaches To Womentioning

confidence: 99%

“…Among them, a systematic approach to lower bounds (which focuses on quadratic cases) is presented in by Arjevani et al in [1], a systematic use of control theory (via integral quadratic constraints) for developing upper bounds is presented by Lessard et al in [26], and the performance estimation approach, which aims at finding worst-case bounds was originally developed in [13] (see also surveys in [11] and [47]). Those methodologies are mostly presented as tools for performing worst-cases analyses (see the numerous examples in [11,19,47,48,50,51]), however, such techniques were also recently used to develop new methods with improved worst-case complexities. Among others, such an approach was used in [13,22] to devise a fixed-step method that attains the best possible worst-case performance for smooth convex minimization [12], and later in [14] to obtain a variant of Kelley's cutting plane method with the best possible worst-case guarantee for non-smooth convex minimization.…”

Section: Links With Systematic and Computer-assisted Approaches To Womentioning

confidence: 99%

“…For many classes of functions F , the condition "S is F -interpolable" can be expressed as a finite set of constraints on the elements of S. In these cases, we refer to this set of constraints as interpolation conditions for the class F . See [48,50] for a list of known interpolation conditions for different classes of functions, along with the corresponding proofs.…”

Section: Basic Definitionsmentioning

confidence: 99%

See 2 more Smart Citations

Efficient first-order methods for convex minimization: a constructive approach

Drori

Taylor

2019

Math. Program.

Self Cite

View full text Add to dashboard Cite

We describe a novel constructive technique for devising efficient first-order methods for a wide range of large-scale convex minimization settings, including smooth, non-smooth, and strongly convex minimization. The technique builds upon a certain variant of the conjugate gradient method to construct a family of methods such that a) all methods in the family share the same worst-case guarantee as the base conjugate gradient method, and b) the family includes a fixed-step first-order method. We demonstrate the effectiveness of the approach by deriving optimal methods for the smooth and non-smooth cases, including new methods that forego knowledge of the problem parameters at the cost of a one-dimensional line search per iteration, and a universal method for the union of these classes that requires a three-dimensional search per iteration. In the strongly convex case, we show how numerical tools can be used to perform the construction, and show that the resulting method offers an improved worst-case bound compared to Nesterov's celebrated fast gradient method. IntroductionConvex optimization plays a central role in many fields of applications, including optimal control, machine learning and signal processing. In particular, when a large number of variables are involved within a convex optimization problem, the use of first-order methods is more and more widespread due to their typically very attractive low computational cost per iteration. This low computational cost comes, however, at a price: first-order methods often suffer from potentially slow convergence speeds, making them appropriate mostly for obtaining low to medium accuracy solutions. Nevertheless, first-order methods remain the methods of choice in many applications and currently receive a lot of attention from the optimization community, which constantly aims at improving them.An effective and fruitful approach used for analyzing and comparing first-order methods is the study of their worst-case behavior through the black-box model. In this setting, methods are only allowed to gain information on the objective through an oracle, which provides the value and the gradient of the objective at selected points.

show abstract

Section: Links With Systematic and Computer-assisted Approaches To Womentioning

confidence: 99%

Section: Links With Systematic and Computer-assisted Approaches To Womentioning

confidence: 99%

Section: Basic Definitionsmentioning

confidence: 99%

See 1 more Smart Citation

Efficient first-order methods for convex minimization: a constructive approach

Drori

Taylor

2019

Math. Program.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The paper is organized as follows: in Section 2, we present the problem statement and the main results; in Section 3, we prove our main results (tight convergence rates for the method and for its exact line search variant); in Section 4, we summarize known and newly derived tight results for the proximal gradient method, that were obtained using the performance estimation framework developed by Drori and Teboulle [9] and the authors (see [10,11]). Finally, we conclude the work in Section 5.…”

Section: Introductionmentioning

confidence: 99%

Exact Worst-Case Convergence Rates of the Proximal Gradient Method for Composite Convex Minimization

Taylor

Hendrickx

Glineur

2018

J Optim Theory Appl

Self Cite

View full text Add to dashboard Cite

We study the worst-case convergence rates of the proximal gradient method for minimizing the sum of a smooth strongly convex function and a non-smooth convex function, whose proximal operator is available.We establish the exact worst-case convergence rates of the proximal gradient method in this setting for any step size and for different standard performance measures: objective function accuracy, distance to optimality and residual gradient norm.The proof methodology relies on recent developments in performance estimation of first-order methods, based on semidefinite programming. In the case of the proximal gradient method, this methodology allows obtaining exact and non-asymptotic worst-case guarantees, that are conceptually very simple, although apparently new.On the way, we discuss how strong convexity can be replaced by weaker assumptions, while preserving the corresponding convergence rates. We also establish that the same fixed step size policy is optimal for all three performance measures. Finally, we extend recent results on the worst-case behavior of gradient descent with exact line search to the proximal case.

show abstract

“…Since M J RK, the computational complexity of BCD-Net is dominated by forward and back projections performed in the MBIR modules. To reduce the M J factor, one can investigate faster optimization methods (e.g., proximal optimized gradient method (POGM) [13]) with ordered subsets. Applying these techniques can reduce the M J factor to (M/G)J , where G is the number of subsets and the number of POGM iterations J < J (e.g., J = (1/ √ 2)J) due to faster convergence rates of POGM over APG.…”

Section: Computational Complexitymentioning

confidence: 99%

BCD-Net for Low-Dose CT Reconstruction: Acceleration, Convergence, and Generalization

Chun

Zheng

Long

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Obtaining accurate and reliable images from low-dose computed tomography (CT) is challenging. Regression convolutional neural network (CNN) models that are learned from training data are increasingly gaining attention in low-dose CT reconstruction. This paper modifies the architecture of an iterative regression CNN, BCD-Net, for fast, stable, and accurate low-dose CT reconstruction, and presents the convergence property of the modified BCD-Net. Numerical results with phantom data show that applying faster numerical solvers to modelbased image reconstruction (MBIR) modules of BCD-Net leads to faster and more accurate BCD-Net; BCD-Net significantly improves the reconstruction accuracy, compared to the state-of-the-art MBIR method using learned transforms; BCD-Net achieves better image quality, compared to a state-of-the-art iterative NN architecture, ADMM-Net. Numerical results with clinical data show that BCD-Net generalizes significantly better than a state-of-the-art deep (non-iterative) regression NN, FBP-ConvNet, that lacks MBIR modules. The authors indicated by asterisks ( * ) equally contributed to this work. Corresponding author: Yong Long (email: yong.long@sjtu.edu.cn). This paper has supplementary document. The prefix "S" indicates the numbers in figure and section in the supplementary document. arXiv:1908.01287v1 [eess.IV] 4 Aug 2019 2 I.Y. Chun and X. Zheng et al.solving large-scale inverse problems in imaging, the first scheme is limited in training CNNs from large-scale images; the second scheme does not effectively remove complicated noise features; and the third scheme has limited benefits when applied to convolutional layers.An alternative way to regulate overfitting of regression CNNs in inverse imaging problems is combining them with model-based image reconstruction (MBIR) that considers imaging physics or image formation models, and noise statistics in measurements. is an iterative regression CNN that generalizes a block coordinate descent (BCD) MBIR method using learned convolutional regularizers [5]. Each layer (or iteration) of BCD-Net consists of image denoising and MBIR modules. In particular, the denoising modules use layer-wise regression CNNs to effectively remove layer-dependent noise features. Many existing works can be viewed as a special case of BCD-Net. For example, RED [11] and MoDL [1] are special cases of BCD-Net, because they use identical image denoising modules across layers or only consider quadratic data-fidelity terms (e.g., the first term in (P1)) in their MBIR modules.This paper modifies BCD-Net that uses convolutional autoencoders in its denoising modules [4], and applies the modified BCD-Net to low-dose CT reconstruction. First, for fast CT reconstruction, we apply the Accelerated Proximal Gradient method using a Majorizer (APG-M), e.g., FISTA [2], to MBIR modules using the statistical CT data-fidelity term. Second, this paper provides the sequence convergence guarantee of BCD-Net when applied to low-dose CT reconstruction. Third, it investigates the generalization ca...

show abstract

Exact Worst-Case Performance of First-Order Methods for Composite Convex Optimization

Cited by 88 publications

References 33 publications

Efficient first-order methods for convex minimization: a constructive approach

Efficient first-order methods for convex minimization: a constructive approach

Exact Worst-Case Convergence Rates of the Proximal Gradient Method for Composite Convex Minimization

BCD-Net for Low-Dose CT Reconstruction: Acceleration, Convergence, and Generalization

Contact Info

Product

Resources

About