Optimal Convergence Rates for the Proximal Bundle Method

Abstract

We study convergence rates of the classic proximal bundle method for a variety of nonsmooth convex optimization problems. We show that, without any modification, this algorithm adapts to converge faster in the presence of smoothness or a Hölder growth condition. Our analysis reveals that with a constant stepsize, the bundle method is adaptive, yet it exhibits suboptimal convergence rates. We overcome this shortcoming by proposing nonconstant stepsize schemes with optimal rates. These schemes use function information such as growth constants, which might be prohibitive in practice. We complete the paper with a new parallelizable variant of the bundle method that attains near-optimal rates without prior knowledge of function parameters. These results improve on the limited existing convergence rates and provide a unified analysis approach across problem settings and algorithmic details. Numerical experiments support our findings and illustrate the effectiveness of the parallel bundle method.

Survey Descent: A Multipoint Generalization of Gradient Descent for Nonsmooth Optimization

“…However, relative to both null and serious steps, priors works analyzing the convergence of bundle methods (see, for example, Kiwiel (2000); Du and Ruszczynski (2017); Diaz and Grimmer (2021)) have only derived sublinear guarantees for global convergence-even when the objective is strongly convex. In contrast, we show in this paper that the Survey Descent iteration, at least in the case of a strongly convex, max-of-smooth function, achieves a local linear convergence rate.…”

“…Juditsky and Nemirovski (2011)). Within the well-studied realm of first-order bundle methods, in particular, theoretical guarantees have remained sublinear relative to the total number of null and serious steps (Kiwiel, 2000;Du and Ruszczynski, 2017;Diaz and Grimmer, 2021)-even in the presence of desirable properties such as δ-strongly convex objectives. For comparison, in the smooth setting, GD (and its projected and proximal variants) possesses well-recognized theoretical guarantees of linear convergence on L-smooth and δ-strongly convex objectives (Beck, 2017, Theorem 10.29).…”

A superlinearly convergent subgradient method for sharp semismooth problems

“…where the penultimate inequality follows from (18) and the last inequality follows from P ker(A i ) ⊥ (y i − ŷ0 ) ≤ γ y i − ŷi and the bound y i − ŷi ≤ y i − ŷ0 . Rearranging, we arrive at the desired conclusion.…”

“…Bundle methods often perform well in practice and their convergence/complexity theory is understood in several settings [23,25,33,38,39,47,54,64]. Most relevantly for this work, on sharp convex functions, variants of the bundle method converge superlinearly relative to the number of serious steps [50] and converge linearly relative to both serious and null steps [18].…”

A unified analysis of a class of proximal bundle methods for solving hybrid convex composite optimization problems

“…for a large range of prox stepsizes λ. Since 2C-PB and MC-PB methods do not rely on (L f , M f , µ), a sharper iteration-complexity bound can be obtained for them by replacing µ and (M f , L f ) in (5) by μ and ( Mf , Lf ), respectively, where μ is the largest µ such that h − µ • 2 /2 is convex and ( Mf , Lf ) is the unique pair which minimizes M 2 f + εL f over the set of pairs (M f , L f ) satisfying the (M f , L f )-hybrid condition of f ′ . Moreover, even though this sharper complexity bound can not be shown for 1C-PB, Section 5 presents an adaptive version of this variant where τ in (3), instead of being chosen as a function of (L f , M f , µ), is adaptively searched so as to satisfy a key inequality condition.…”