The fastest L1,oo prox in the west

Abstract

Proximal operators are of particular interest in optimization problems dealing with non-smooth objectives because in many practical cases they lead to optimization algorithms whose updates can be computed in closed form or very efficiently. A well-known example is the proximal operator of the vector 1 norm, which is given by the soft-thresholding operator. In this paper we study the proximal operator of the mixed 1,∞ matrix norm and show that it can be computed in closed form by applying the well-known soft-thresholding operator to each column of the matrix. However, unlike the vector 1 norm case where the threshold is constant, in the mixed 1,∞ norm case each column of the matrix might require a different threshold and all thresholds depend on the given matrix. We propose a general iterative algorithm for computing these thresholds, as well as two efficient implementations that further exploit easy to compute lower bounds for the mixed norm of the optimal solution. Experiments on large-scale synthetic and real data indicate that the proposed methods can be orders of magnitude faster than state-of-the-art methods.

Efficient projection algorithms onto the weighted ℓ1 ball

“…The projection onto the ℓ 1,∞ ball has gain interest in the last years [29,30,31,32]. The main reasons being its efficiency to enforce sparsity and most importantly to often increase accuracy.…”

“…As detailed in Section 2 of [32], the projection onto the ℓ 1,∞ norm ball can be used to compute the proximity operator of the dual norm, which is the ℓ ∞,1 norm:…”

“…The ℓ 1,∞ norm is of particular interest because, compared to other norms, it is able to set a whole set of columns to zero, instead of spreading zeros as done by the ℓ 1 or ℓ 1,2 norms. This makes it particularly interesting for machine learning applications, and this is why many projection algorithms have been proposed [29,30,31,32].…”

Dictionary-Based Low-Rank Approximations and the Mixed Sparse Coding Problem

“…To again leverage FISTA to solve Problem (25) and produce a support for a solution of MSC requires to compute the proximal operator of the regularization term. For the ℓ 1,1 norm, the proximal operator has been shown to be computable exactly with little cost using a bisection search [71][72][73]. In this work I used the low-level implementation of [72] 6 .…”

“…For the ℓ 1,1 norm, the proximal operator has been shown to be computable exactly with little cost using a bisection search [71][72][73]. In this work I used the low-level implementation of [72] 6 . The resulting Mixed-FISTA algorithm is very similar to Algorithm 2 but using the ℓ 1,1 proximal operator instead of soft-thresholding, and its pseudocode is therefore differed to the Supplementary Material.…”