SABRINA: A Stochastic Subspace Majorization-Minimization Algorithm

Abstract

A wide class of problems involves the minimization of a coercive and differentiable function F on R N whose gradient cannot be evaluated in an exact manner. In such context, many existing convergence results from standard gradientbased optimization literature cannot be directly applied and robustness to errors in the gradient is not necessarily guaranteed. This work is dedicated to investigating the convergence of Majorization-Minimization (MM) schemes when stochastic errors affect the gradient terms. We introduce a general stochastic optimization framework, called SABRINA (StochAstic suBspace majoRIzation-miNimization Algorithm) that encompasses MM quadratic schemes possibly enhanced with a subspace acceleration strategy. New asymptotical results are built for the stochastic process generated by SABRINA. Two sets of numerical experiments in the field of machine learning and image processing are presented to support our theoretical results and illustrate the good performance of SABRINA with respect to state-of-the-art gradient-based stochastic optimization methods.

Block delayed Majorize-Minimize subspace algorithm for large scale image restoration *

“…This strategy has been initially introduced for full-batch MM algorithms (i.e., without any block coordinate strategy) in [17]. Convergence analysis can be found in [18,16,21,20,19] under various situations. We recently extended this strategy to cope with block coordinate updates with the form of (B2M) [33], leading to the B2MS (Block MM Subspace) scheme that we present hereafter.…”

“…Several choices for the subspace matrix are discussed in [17,21,19]. Intensive comparisons in the fields of inverse problems, image processing and machine learning (e.g., [34,16]), have shown the superiority of the socalled memory gradient subspace which seems to reach the best compromise between simplicity and efficiency. In the context of (B2MS), this amounts to defining, for every k P N, the memory gradient matrix D k "…”

A Kurdyka-Lojasiewicz property for stochastic optimization algorithms in a non-convex setting

“…Actually, almost-sure convergence based on sequence (x k ) k∈N for stochastic gradient schemes remains scarcely studied in the non-convex case. Recent results in [25,14] proposed an alternative proof technique without KL inequality by constraining the topology of the stationary points of F .…”

An Accelerated Variance Reduction Stochastic ADMM for Nonconvex Nonsmooth Optimization

“…Many effective functions satisfy the KL property, including but not limited to ∥x∥ 1 , ∥Ax − b∥ 2 , log-exp, and the logistic loss function ψ(t) = log 1 + e −t . For more examples, readers can refer to [28], [29], [30], [31], and [32](see Page 919). Therefore, the potential energy function defined in (15) can easily satisfy the KL property.…”

“…It is important to acknowledge that the implementation of the KL technique displays slight differences between stochastic and deterministic algorithms. For further details, readers are encouraged to refer to [30], [31].…”