On Lipschitz optimization based on gray-box piecewise linearization

Abstract

We address the problem of minimizing objectives from the class of piecewise differentiable functions whose nonsmoothness can be encapsulated in the absolute value function. They possess local piecewise linear approximations with a discrepancy that can be bounded by a quadratic proximal term. This overestimating local model is continuous but generally nonconvex. It can be generated in its absnormal form by a minor extension of standard algorithmic differentiation tools. Here we demonstrate how the local model can be minimized by a bundle-type method, which benefits from the availability of additional gray-box information via the absnormal form. In the convex case our algorithm realizes the consistent steepest descent trajectory for which finite convergence was established earlier, specifically covering counterexamples where steepest descent with exact line-search famously fails. The analysis of the abs-normal representation and the design of the optimization algorithm are geared toward the general case, whereas the convergence proof so far covers only the convex case.

Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

“…Let us mention [29] which developed piecewise linear approximation for functions which can be expressed using absolute value, min or max operators. This approach led to successful algorithmic developments [30] but may suffer from a high computational complexity and a lack of versatility (the Euclidean norm cannot be dealt with within this framework). Another attempt using the same model of branching programs was described in [33] where a qualification assumption is used to compute Clarke generalized derivatives automatically 6 .…”

Relaxing Kink Qualifications and Proving Convergence Rates in Piecewise Smooth Optimization

“…However, we will also consider frequently the situation where σ varies over all possibilities {−1, 0, 1} s . As observed already in [10] for the nonlinear case, the limiting gradients of ϕ in the vicinity of x are given by (8) g…”

“…This partial ordering of the signature vectors was already used in [10]. Like in the piecewise linear case we can find that the closureS σ of any S σ is contained in the extended closureŜ…”

“…for all x ∈ S σ (10) and ϕ σ is one of finitely many differentiable selection functions in the sense of Scholtes [16]. At a given point x, the nonsmoothness of the target function ϕ is caused by the so-called active switching variables z i (x) = 0 for 1 ≤ i ≤ s. We collect them in the active switch set…”

“…One version called LiPsMin of the SPLOP algorithm discussed in Sec. 1 is analyzed in [4,10]. Its C++ based optimization is used to generate the numerical results presented in this section.…”