Survey Descent: A Multipoint Generalization of Gradient Descent for Nonsmooth Optimization

Abstract

For strongly convex objectives that are smooth, the classical theory of gradient descent ensures linear convergence relative to the number of gradient evaluations. An analogous nonsmooth theory is challenging: even when the objective is smooth at every iterate, the corresponding local models are unstable, and traditional remedies need unpredictably many cutting planes. We instead propose a multipoint generalization of the gradient descent iteration for local optimization. While designed with general objectives in mind, we are motivated by a "maxof-smooth" model that captures subdifferential dimension at optimality. We prove linear convergence when the objective is itself max-of-smooth, and experiments suggest a more general phenomenon.

A nearly linearly convergent first-order method for nonsmooth functions with quadratic growth

“…A popular class of decomposable objectives arise from pointwise maxima of smooth functions that satisfy an affine independence property. For example, this class was considered in the work of Han and Lewis [25]. As an immediate corollary of Theorem 3.3, we show that such functions satisfy Assumption A.…”

“…The local rate holds under a key structural assumption -Assumption A -which formalizes the concept of typical structure and mirrors the structure of the simple function considered in Section 1.3.1. While we formally describe Assumption A in Section 3, for now, we mention that it holds for max-of-smooth and properly C p decomposable functions, provided the local minimizer x is in fact a strong local minimizer that satisfies a strict complementarity condition; this class that includes the max-of-smooth setting considered in [25]. Assumption A also holds for generic linear tilts of semialgebraic functions: if f is semialgebraic, then for a full Lebesgue measure set of w ∈ R d , Assumption A holds at every local minimizer x of the tilted function f w : x → f (x) + w x.…”

“…Second, from the statement of the theorems, we see that the neighborhood of local nearly linear convergence, B δNTD (x), depends solely on f . Before turning to our second experiment, let us briefly mention two alternative methods -Prox-linear [10,22,23,44,47] and Survey Descent [25] -which could be applied to this problem. In order to explain these algorithms, let us write f = max i=1,...,m {f i }, where the f i are the quadratic function from (7.1).…”

“…. , m. In [25], Han and Lewis study linear convergence of Survey Descent on max-of-smooth functions under the conditions of Corollary 3.4. Given a survey S, they show that the updated survey S + geometrically improves on S (in an appropriate sense) whenever the following conditions are satisfied: (i) all elements of the survey S are near x; (i) the survey S is valid, meaning there exists a permutation a on [m] such that…”

“…These methods have excellent empirical performance, but a complete account of their inner-loop complexity remains elusive. On the other hand, in a recent breakthrough, Han and Lewis proposed a first-order method -Survey Descent -that converges linearly on certain strongly convex max-of-smooth objectives, stepping beyond the classical smooth setting [25]. The method shows favorable performance beyond the max-of-smooth class, e.g., on certain eigenvalue optimization problems, but no theoretical justification for this success is available.…”