Nearly optimal linear embeddings into very low dimensions

Abstract

Abstract-We propose algorithms for constructing linear embeddings of a finite dataset V ⊂ R d into a k-dimensional subspace with provable, nearly optimal distortions. First, we propose an exhaustive-search-based algorithm that yields a k-dimensional linear embedding with distortion at most opt(k)+δ, for any δ > 0 where opt(k) is the smallest achievable distortion over all possible orthonormal embeddings. This algorithm is space-efficient and can be achieved by a single pass over the data V . However, the runtime of this algorithm is exponential in k. Second, we propose a convexprogramming-based algorithm that yields an O (k/δ)-dimensional orthonormal embedding with distortion at most (1 + δ) opt(k). The runtime of this algorithm is polynomial in d and independent of k. Several experiments demonstrate the benefits of our approach over conventional linear embedding techniques, such as principal components analysis (PCA) or random projections.

NuMax: A Convex Approach for Learning Near-Isometric Linear Embeddings

“…However, this seems to an extremely challenging analytical problem for a generic point set X . For some initial progress in this direction (albeit under somewhat more restrictive settings), see the recent results by [32] and [33]. The main question is to verify the efficiency of the convex relaxation (4), which is essentially an SDP with rank-1 constraints (specified by the secant set S(X )).…”

“…Since the appearance of a preliminary version of this manuscript, several works have pursued similar goals of learning norm-preserving linear embeddings using optimization methods. The authors of [32] discuss the specialized problem of learning orthonormal linear embeddings, and develop polynomial-time algorithms with provable approximation guarantees. The authors of [33] propose learning embeddings under a Frobenius norm constraint, and propose a different optimization approach with provable guarantees.…”

Practical Algorithms for Learning Near-Isometric Linear Embeddings

“…This framework deterministically produces a near-isometric linear embedding. Other algorithmic approaches for finding near-isometric linear embeddings are also described in [4,6,19]. …”

Metric learning with rank and sparsity constraints

“…It is also possible to add a constraint on the Frobenius norm of A directly. Note that the approach in [4] solves a convex relaxation of (6), where the rank constraint is replaced by a nuclear norm constraint. That solution works in the N × N space and requires eigendecompositions.…”

“…Hence, the learned metric creates storage as well as computational bottlenecks. As a result, several works [2,3,5,6] consider learning rank-constrained metrics. Unfortunately, enforcing rank constraints on the already computationally challenging MLP (1) makes it non-convex.…”