NuMax: A Convex Approach for Learning Near-Isometric Linear Embeddings

Abstract

We propose a novel framework for the deterministic construction of linear, near-isometric embeddings of a finite set of data points. Given a set of training points X ⊂ R N , we consider the secant set S(X ) that consists of all pairwise difference vectors of X , normalized to lie on the unit sphere. We formulate an affine rank minimization problem to construct a matrix Ψ that preserves the norms of all the vectors in S(X ) up to a distortion parameter δ. While affine rank minimization is NP-hard, we show that this problem can be relaxed to a convex formulation that can be solved using a tractable semidefinite program (SDP). In order to enable scalability of our proposed SDP to very large-scale problems, we adopt a twostage approach. First, in order to reduce compute time, we develop a novel algorithm based on the Alternating Direction Method of Multipliers (ADMM) that we call Nuclear norm minimization with Max-norm constraints (NuMax) to solve the SDP. Second, we develop a greedy, approximate version of NuMax based on the column generation method commonly used to solve large-scale linear programs.We demonstrate that our framework is useful for a number of signal processing applications via a range of experiments on large-scale synthetic and real datasets.

Metric learning with rank and sparsity constraints

“…In the first set of experiments we investigate how δ varies with the rank of the matrix we learn using both a set of synthetic data and a set of images of motorbikes [23]. The synthetic data set is the manifold data set used in [4], composed of translating white squares in a black background. We generate manifold images sizes of 40 × 40 pixels and resize grayscale images of the motorbikes to also be 40 × 40 pixels, resulting in points of dimension N = 1600.…”

“…In general, the MLP (1) considers relationships between points based on their pairwise distances. Hence, we would require that the metric B preserves the pairwise distances of the points in D up to a distortion parameter δ as in (3) to yield a more stringent constraint for (1) while ignoring this requirement for points in S. However, we set up a symmetric problem which then uses RIP in the manner of [4,5] that can be adjusted depending on the individual application. In this symmetric formulation, using secant vectors, (3) simplifies to |v…”

“…It is also possible to add a constraint on the Frobenius norm of A directly. Note that the approach in [4] solves a convex relaxation of (6), where the rank constraint is replaced by a nuclear norm constraint. That solution works in the N × N space and requires eigendecompositions.…”

“…Moreover, as the number of variables is quadratic with the number of features, even the most basic gradient-only approaches do not scale well [1,4].…”

“…To illustrate our algorithm, we use a symmetric and non-smooth version of the MLP (1) in the manner of [4,5]. We then use the Nesterov smoothing approach to obtain a smooth cost that has approximation guarantees to the original problem, followed by Burer-Monteiro splitting [10] with quasi-Newton enhancements.…”