To convexify or not? Regression with clustering penalties on graphs

Abstract

Abstract-We consider minimization problems that are compositions of convex functions of a vector x ∈ R N with submodular set functions of its support (i.e., indices of the non-zero coefficients of x). Such problems are in general difficult for large N due to their combinatorial nature. In this setting, existing approaches rely on "convexifications" of the submodular set function based on the Lovász extension for tractable approximations. In this paper, we first demonstrate that such convexifications can fundamentally change the nature of the underlying submodular regularization. We then provide a majorizationminimization framework for the minimization of such composite objectives. For concreteness, we use the Ising model to motivate a submodular regularizer, establish the total variation semi-norm as its Lovász extension, and numerically illustrate our new optimization framework.

Structured Sparsity: Discrete and Convex Approaches

“…In the discrete setting, (26) is preserved in an attempt to faithfully encode the discrete model at hand. While such criterion seems cumbersome to solve, it has favorable properties that lead to polynomial solvability, irrespective of its combinatorial nature: there are efficient combinatorial algorithms that provide practical solutions to overcome this bottleneck and are guaranteed to converge in polynomial time [43]; see Section 7.1.…”

“…We can also view the Ising penalty as a cut function: R ISING (S) indeed just counts the number of edges that are cut by the set S. Cut functions with appropriate graphs and weights actually capture a large subset of submodular functions [72,78]. Since R ISING is symmetric, its convexification is just its Lovász extension, as discussed above, which is shown [43] to be the anisotropic discrete Total Variation semi-norm…”

“…If the function g is nonsmooth and possesses a closed form proximity operator, then general primal-dual methods such as [29,60] have been known as powerful tools to tackle problem (44). Since g is nonsmooth, one can also use the primal-dual subgradient method [106] as well as the prox-method proposed in [102] for solving (43). An alternative approach is to combine primal-dual optimization methods and smoothing technique as done in [104,105] for solving (44).…”

Map estimation for Bayesian mixture models with submodular priors

“…Here, we aim to efficiently compute numerically good approximations to the MAP estimator. Given our assumptions, the objective function in (1) can be iteratively minimized by the majorizationminimization scheme of Algorithm 1, see also [10]. The main idea is to majorize the continuous part L y (x) + G(x|s) at each iteration by a modular upper bound, and then solve the resulting SFM.…”

“…However, the presence of the discrete component R(S) in our model renders the optimization difficult. We present an extension of the efficient Majorization-Minimization algorithm introduced in [10] that iteratively maximizes the log-posterior log p(x, s|y), with guaranteed convergence. Our numerical results show that the proposed algorithm can take full advantage of all available prior information on the signal, while for non-truly sparse signals, state-of-the-art methods are capable of leveraging only a part of it.…”