Smoothing and worst-case complexity for direct-search methods in nonsmooth optimization

Abstract

In the context of the derivative-free optimization of a smooth objective function, it has been shown that the worst case complexity of direct-search methods is of the same order as the one of steepest descent for derivative-based optimization, more precisely that the number of iterations needed to reduce the norm of the gradient of the objective function below a certain threshold is proportional to the inverse of the threshold squared.Motivated by the lack of such a result in the non-smooth case, we propose, analyze, and test a class of smoothing direct-search methods for the unconstrained optimization of nonsmooth functions. Given a parameterized family of smoothing functions for the non-smooth objective function dependent on a smoothing parameter, this class of methods consists of applying a direct-search algorithm for a fixed value of the smoothing parameter until the step size is relatively small, after which the smoothing parameter is reduced and the process is repeated.One can show that the worst case complexity (or cost) of this procedure is roughly one order of magnitude worse than the one for direct search or steepest descent on smooth functions.The class of smoothing direct-search methods is also showed to enjoy asymptotic global convergence properties. Some preliminary numerical experiments indicates that this approach leads to better values of the objective function, pushing in some cases the optimization further, apparently without an additional cost in the number of function evaluations.

Worst case complexity of direct search under convexity

“…Cartis, Gould, and Toint [2] have derived a WCC bound of O(n 2 −3/2 ) for their adaptive cubic overestimation algorithm when using finite differences to approximate derivatives. In the non-smooth case, using smoothing techniques, both Garmanjani and Vicente [6] and Nesterov [12] established a WCC bound of approximately O( −3 ) iterations for their zero-order methods, where the threshold refers now to the gradient of a smoothed version of the original function.…”

Worst-Case Complexity of Smoothing Quadratic Regularization Methods for Non-Lipschitzian Optimization

“…They prove that it takes at most O( −2 ) iterations to reduce a first order criticality measure below in a first order trust region method or a quadratic regularization method, where the worst-case complexity result is the same in order as the function evaluation complexity of steepest descent methods applied to the case that Φ h is differentiable. In [19], Garmanjani and Vicente propose a smoothing direct search (DS) algorithm based on smoothing techniques and derivative-free methods to solve a general unconstrained nonsmooth nonconvex, Lipschitzian minimization problem. The smoothing DS algorithm can be seen as a zero order method, where the gradient is not calculated in the algorithm.…”

“…In this paper, we present a smoothing quadratic regularization (SQR) algorithm for solving (1.1) with the worst-case complexity estimation. The SQR algorithm uses the smoothing functions [2,10,19,26,30] and regularization methods [6,7,8,28]. At each iteration, the SQR algorithm solves a strongly convex quadratic minimization problem with a diagonal Hessian matrix, which has a simple closed-form solution that is inexpensive to calculate.…”

“…As expectation, when applying the gradient off , the modified SQR Downloaded 01/13/14 to 158.132.161.52. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php algorithm for locally Lipschitz continuous minimization has a better complexity result than the smoothing DS algorithm proposed in [19], in which the gradient information is not used. Note that many penalty functions cannot be written as h(c(x)) in (1.2), for example, the logistic penalty function, ϕ(|x i |) = log(1 + α|x i |).…”

“…Similar to the proof idea of Theorem 4.4, the SQR 1 algorithm takes at most O( −3 ) iterations to reduce ∇f (x, μ) ∞ ≤ and μ ≤ , while the DS algorithm in [19] needs to take at most O( −3 log −1 ) iterations. For given ∈ (0, 1], it is difficult to verify the inequality…”