Convergence Analysis of Perturbed Feasible Descent Methods

Solodov, M. V.

doi:10.1023/a:1022602123316

Cited by 14 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even the reverse setting is generalized in the non-convex setting [5,11], namely where the backward-step is performed on a non-smooth non-convex function.As the amount of data to be processed is growing and algorithms are supposed to exploit all the data in each iteration, inexact methods become interesting, though we do not consider erroneous estimates in this paper. Forward-backward splitting schemes also seem to work for non-convex problems with erroneous estimates [44,43]. A mathematical analysis of inexact methods can be found, e.g., in [14,5], but with the restriction that the method is explicitly required to decrease the function values in each iteration.…”

mentioning

confidence: 99%

iPiano: Inertial Proximal Algorithm for Nonconvex Optimization

Ochs¹,

Chen²,

Brox³

et al. 2014

SIAM J. Imaging Sci.

364

373

View full text Add to dashboard Cite

Abstract. In this paper we study an algorithm for solving a minimization problem composed of a differentiable (possibly non-convex) and a convex (possibly non-differentiable) function. The algorithm iPiano combines forward-backward splitting with an inertial force. It can be seen as a non-smooth split version of the Heavy-ball method from Polyak. A rigorous analysis of the algorithm for the proposed class of problems yields global convergence of the function values and the arguments. This makes the algorithm robust for usage on non-convex problems. The convergence result is obtained based on the Kurdyka-Lojasiewicz inequality. This is a very weak restriction, which was used to prove convergence for several other gradient methods. First, an abstract convergence theorem for a generic algorithm is proved, and, then iPiano is shown to satisfy the requirements of this theorem. Furthermore, a convergence rate is established for the general problem class. We demonstrate iPiano on computer vision problems: image denoising with learned priors and diffusion based image compression.Key words. non-convex optimization, Heavy-ball method, inertial forward-backward splitting, Kurdyka-Lojasiewicz inequality, proof of convergence 1. Introduction. The gradient method is certainly one of the most fundamental but also one of the most simple algorithms to solve smooth convex optimization problems. In the last decades, the gradient method has been modified in many ways. One of those improvements is to consider so-called multi-step schemes [38,35]. It has been shown that such schemes significantly boost the performance of the plain gradient method. Triggered by practical problems in signal processing, image processing and machine learning, there has been an increased interest in so-called composite objective functions, where the objective function is given by the sum of a smooth function and a non-smooth function with an easy to compute proximal map. This initiated the development of the so-called proximal gradient or forward-backward method [28], that combines explicit (forward) gradient steps w.r.t. the smooth part with proximal (backward) steps w.r.t. the non-smooth part.In this paper, we combine the concepts of multi-step schemes and the proximal gradient method to efficiently solve a certain class of non-convex, non-smooth optimization problems. Although, the transfer of knowledge from convex optimization to non-convex problems is very challenging, it aspires to find efficient algorithms for certain non-convex problems. Therefore, we consider the subclass of non-convex problems

show abstract

mentioning

confidence: 99%

iPiano: Inertial Proximal Algorithm for Nonconvex Optimization

Ochs¹,

Chen²,

Brox³

et al. 2014

SIAM J. Imaging Sci.

364

373

View full text Add to dashboard Cite

show abstract

“…At the same time, we also prove that either (13) or (15) is true with Wolfe or Armijo or Goldstein linesearch on the uniformly convex function. In doing so, we remove various boundedness conditions such as boundedness from below of f (·), boundedness of x k , etc.…”

Section: Introductionmentioning

confidence: 61%

Global Convergence of the Dai-Yuan Conjugate Gradient Method with Perturbations

Wang

2007

Jrl Syst Sci & Complex

View full text Add to dashboard Cite

In this paper, the authors propose a class of Dai-Yuan (abbr. DY) conjugate gradient methods with linesearch in the presence of perturbations on general function and uniformly convex function respectively. Their iterate formula is x k+1 = x k + α k (s k + ω k ), where the main direction s k is obtained by DY conjugate gradient method, ω k is perturbation term, and stepsize α k is determined by linesearch which does not tend to zero in the limit necessarily. The authors prove the global convergence of these methods under mild conditions. Preliminary computational experience is also reported.

show abstract

“…Then, assertion (16) follows from the optimality condition for (7). We proceed to prove the last assertion.…”

Section: Convergence Propertiesmentioning

confidence: 82%

“…where d k is computed using the solution of (14) [dual to (7), see Lemma 3.1]. Relations (4), (5) essentially constitute the usual stopping test in bundle methods; see e.g.…”

Section: Bundle Methods With Inexact Datamentioning

confidence: 99%

On Approximations with Finite Precision in Bundle Methods for Nonsmooth Optimization

Solodov

2003

Journal of Optimization Theory and Applications

Self Cite

View full text Add to dashboard Cite

Abstract. We consider the proximal form of a bundle algorithm for minimizing a nonsmooth convex function, assuming that the function and subgradient values are evaluated approximately. We show how these approximations should be controlled in order to satisfy the desired optimality tolerance. For example, this is relevant in the context of Lagrangian relaxation, where obtaining exact information about the function and subgradient values involves solving exactly a certain optimization problem, which can be relatively costly (and as we show, in any case unnecessary). We show that approximation with some finite precision is sufficient in this setting and give an explicit characterization of this precision. Alternatively, our result can be viewed as a stability analysis of standard proximal bundle methods, as it answers the following question: for a given approximation error, what kind of approximate solution can be obtained and how does it depend on the magnitude of the perturbation?

show abstract

Convergence Analysis of Perturbed Feasible Descent Methods

Cited by 14 publications

References 27 publications

iPiano: Inertial Proximal Algorithm for Nonconvex Optimization

iPiano: Inertial Proximal Algorithm for Nonconvex Optimization

Global Convergence of the Dai-Yuan Conjugate Gradient Method with Perturbations

On Approximations with Finite Precision in Bundle Methods for Nonsmooth Optimization

Contact Info

Product

Resources

About