Smooth minimization of non-smooth functions

Nesterov, Yurii

doi:10.1007/s10107-004-0552-5

Cited by 2,117 publications

(2,292 citation statements)

References 7 publications

Supporting

Mentioning

2,254

Contrasting

Unclassified

Order By: Relevance

“…First, when all φ ℓ are smooth, so is φ, in contrast to the objective in (7) which typically is nonsmooth; this makes the saddle point reformulation better suited for processing by first order algorithms. Starting with the breakthrough paper of Nesterov [12], this phenomenon, in its general form, is utilized in the fastest known so far first order algorithms for "well-structured" nonsmooth convex programs. Second, in the stochastic case, stochastic oracles providing unbiased estimates of the first order information on φ i oracles, while not induce a similar oracle for the objective of (7), do induce such an oracle for the v.i.…”

Section: Composite Optimization Problem and Its Saddle Point Reformulmentioning

confidence: 99%

“…In fact, the "best approximations" available are given by Robust Stochastic Approximation (see [4] and references therein) with the guaranteed rate of convergence O(1) L+M +σ √ t and extra-gradient-type algorithms for solving deterministic monotone v.i. 's with Lipschitz continuous operators (see [9,[12][13][14]), which attain the accuracy O(1) L t in the case of M = σ = 0 or O(1) M √ t when L = σ = 0. The goal of this paper is to demonstrate that a specific Mirror-Prox algorithm [9] for solving monotone v.i.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Solving Variational Inequalities with Stochastic Mirror-Prox Algorithm

2011

View full text Add to dashboard Cite

In this paper we consider iterative methods for stochastic variational inequalities (s.v.i.) with monotone operators. Our basic assumption is that the operator possesses both smooth and nonsmooth components. Further, only noisy observations of the problem data are available. We develop a novel Stochastic Mirror-Prox (SMP) algorithm for solving s.v.i. and show that with the convenient stepsize strategy it attains the optimal rates of convergence with respect to the problem parameters. We apply the SMP algorithm to Stochastic composite minimization and describe particular applications to Stochastic Semidefinite Feasibility problem and deterministic Eigenvalue minimization.1. Introduction. Variational inequalities with monotone operators form a convenient framework for unified treatment (including algorithmic design) of problems with "convex structure", like convex minimization, convexconcave saddle point problems and convex Nash equilibrium problems. In this paper we utilize this framework to develop first order algorithms for stochastic versions of the outlined problems, where the precise first order information is replaced with its unbiased stochastic estimates. This situation arises naturally in convex Stochastic Programming, where the precise first order information is unavailable (see examples in section 4). In some situations, e.g. those considered in [4, Section 3.3] and in Section 4.4, where passing from available, but relatively computationally expensive precise first order information to its cheap stochastic estimates allows to accelerate the solution process, with the gain from randomization growing progressively with problem's sizes.Our "unifying framework" is as follows. Let Z be a convex compact set in Euclidean space E with inner product ·, · , · be a norm on E (not

show abstract

Section: Composite Optimization Problem and Its Saddle Point Reformulmentioning

confidence: 99%

mentioning

confidence: 99%

Solving Variational Inequalities with Stochastic Mirror-Prox Algorithm

2011

View full text Add to dashboard Cite

show abstract

“…Finally, smooth approximations to a wide class of non-smooth convex functions are available thanks to the recent developments in the non-smooth convex optimization literature (see e.g. Nesterov, 2005;Beck & Teboulle, 2012). Using these results, we provide functional forms of smooth approximations to some of the commonly used test statistics.…”

Section: Regularization Of Test Statisticsmentioning

confidence: 99%

“…While the idea of regularization appears in some other contexts, such as Haile & Tamer (2003), Chernozhukov, Kocatulum & Menzel (2015) and Masten & Poirier (2017), we formally show that this approach has uniform validity in the context of inference with simulated variables. Our regularization method is based on the class of µ-smooth approximations studied in the non-smooth optimization literature (Nesterov, 2005;Beck & Teboulle, 2012). We provide conditions on the choice of approximating functions and regularization parameters that ensure the uniform validity of an inference procedure that combines the proposed regularization scheme with a straightforward bootstrap resampling.…”

Section: Introductionmentioning

confidence: 99%

Moment inequalities in the context of simulated and predicted variables

Kaido¹,

Li²,

Rysman³

2018

View full text Add to dashboard Cite

This paper explores the effects of simulated moments on the performance of inference methods based on moment inequalities. Commonly used confidence sets for parameters are level sets of criterion functions whose boundary points may depend on sample moments in an irregular manner. Due to this feature, simulation errors can affect the performance of inference in non-standard ways. In particular, a (first-order) bias due to the simulation errors may remain in the estimated boundary of the confidence set. We demonstrate, through Monte Carlo experiments, that simulation errors can significantly reduce the coverage probabilities of confidence sets in small samples. The size distortion is particularly severe when the number of inequality restrictions is large. These results highlight the danger of ignoring the sampling variations due to the simulation errors in moment inequality models. Similar issues arise when using predicted variables in moment inequalities models. We propose a method for properly correcting for these variations based on regularizing the intersection of moments in parameter space, and we show that our proposed method performs well theoretically and in practice.

show abstract

“…This type of smoothing has been proposed by many authors for solving convex finite minimax problems, in particular by Bertsekas [7], Ben-Tal and Teboulle [6], Alvarez [1], and Nesterov [18]. This smoothing approach has been also proposed by Polak-Royset-Womersley [20], by Sheu-Wu [27] for finite min-max problems subject to infinitely many linear constraints and, more recently, by Sheu-Lin [26] for continuous min-max problems, motivated by the global approach of Fang-Wu [12] using an integral analog.…”

Section: Introductionmentioning

confidence: 99%

Penalty and Smoothing Methods for Convex Semi-Infinite Programming

2009

View full text Add to dashboard Cite

In this paper we consider min-max convex semi-infinite programming. In order to solve these problems we introduce a unified framework concerning Remez-type algorithms and integral methods coupled with penalty and smoothing methods. This framework subsumes well-known classical algorithms, but also provides some new methods with interesting properties. Convergence of the primal and dual sequences are proved under minimal assumptions.

show abstract

Smooth minimization of non-smooth functions

Cited by 2,117 publications

References 7 publications

Solving Variational Inequalities with Stochastic Mirror-Prox Algorithm

Solving Variational Inequalities with Stochastic Mirror-Prox Algorithm

Moment inequalities in the context of simulated and predicted variables

Penalty and Smoothing Methods for Convex Semi-Infinite Programming

Contact Info

Product

Resources

About