A geometric integration approach to nonsmooth, nonconvex optimisation

Abstract

The optimisation of nonsmooth, nonconvex functions without access to gradients is a particularly challenging problem that is frequently encountered, for example in model parameter optimisation problems. Bilevel optimisation of parameters is a standard setting in areas such as variational regularisation problems and supervised machine learning. We present efficient and robust derivative-free methods called randomised Itoh-Abe methods. These are generalisations of the Itoh-Abe discrete gradient method, a wellknown scheme from geometric integration, which has previously only been considered in the smooth setting. We demonstrate that the method and its favourable energy dissipation properties are well-defined in the nonsmooth setting. Furthermore, we prove that whenever the objective function is locally Lipschitz continuous, the iterates almost surely converge to a connected set of Clarke stationary points. We present an implementation of the methods, and apply it to various test problems. The numerical results indicate that the randomised Itoh-Abe methods are superior to state-of-the-art derivative-free optimisation methods in solving nonsmooth problems while remaining competitive in terms of efficiency.

Inexact Derivative-Free Optimization for Bilevel Learning

“…Bilevel learning with a DFO algorithm was previously considered [16], but there a different DFO method based on discrete gradients was used, and was applied to nonsmooth problems with exact lower-level evaluations. In [16], only up to two parameters were learned, whereas here we demonstrate our approach is capable of learning many more. Our numerical results include examples with up to 64 parameters.…”

“…A by-now common strategy to learn parameters of a variational regularization model from data is bilevel learning, see e.g. [11][12][13][14][15][16][17] and references in [4] . Given labelled data (x i , y i ) i=1,...,n we find parameters θ ∈ Θ ⊂ R m by solving the upper-level The lower-level objective Φ i,θ could be of the form Φ i,θ (x) = D(Ax, y i ) + θR(x) as in (1.1) but we will not restrict ourselves to this special case.…”

“…This requirement is practically infeasible and common algorithms in the literature compute the lower-level solution only to some accuracy, thereby losing any theoretical performance guarantees, see e.g. [11][12][13]16]. One reason for needing exact solutions is to compute the gradient of the upper-level objective using the implicit function theorem [11], which we address by using upper-level solvers which do not require gradient computations.…”

“…Here, we are interested in the particular setting of learning for variational methods (1.2), which has also been considered in [16] where a new DFO algorithm based on discrete gradients has been proposed. In [16] it was assumed that the lower-level problem can be solved exactly such that the bilevel problem can be reduced to a single nonconvex optimization problem. In the present work we lift this stringent assumption.…”

Variational Image Regularization with Euler's Elastica Using a Discrete Gradient Scheme

“…In the case of the Itoh-Abe discrete gradient, the rates have an O(n 1/2 ) dependence on the problem size n; in Lemma 1 we show that the rates can be improved when V has sparsely connected unknowns. A convergence result for Itoh-Abe type discrete gradient schemes applied to non-differentiable, non-convex problems is presented in [27], with applications to parameter estimation problems.…”