Optimal Fast Johnson-Lindenstrauss Embeddings for Large Data Sets

Abstract

We introduce a new fast construction of a Johnson-Lindenstrauss matrix based on the composition of the following two embeddings: A fast construction by the second author joint with Ward [1] maps points into a space of lower, but not optimal dimension. Then a subsequent transformation by a dense matrix with independent entries reaches an optimal embedding dimension.As we show in this note, the computational cost of applying this transform simultaneously to all points in a large data set comes close to the complexity of just reading the data under only very mild restrictions on the size of the data set. Along the way, our construction also yields the least restricted Johnson-Lindenstrauss Transform of order optimal embedding dimension known to date that allows for a fast query step, that is, a fast application to an arbitrary point that is not part of the given data set.

An Introduction to Johnson-Lindenstrauss Transforms

“…More explicitely if 𝑓 1 : R 𝑑 → R 𝑑 ′ and 𝑓 2 : R 𝑑 ′ → R 𝑚 with 𝑚 ≪ 𝑑 ′ ≪ 𝑑 are two JLTs and computing 𝑓 1 (𝑥) is fast, we could hope that computing ( 𝑓 2 • 𝑓 1 )(𝑥) is fast as well as 𝑓 2 only need to handle 𝑑 ′ dimensional vectors and hope that ( 𝑓 2 • 𝑓 1 ) preserves the norm sufficiently well since both 𝑓 1 and 𝑓 2 approximately preserve norms individually. As presented here, the obvious candidate for 𝑓 1 is one of the RIP-based JLDs, which was succesfully applied in [BK17]. In their construction, which we will refer to as GRHD28, 𝑓 1 is the SRHT and 𝑓 2 is the dense Rademacher construction (i.e.…”

“…Let 𝐴 be a 𝑚 × 𝑛 matrix and 𝐵 be a 𝑝 × 𝑞 matrix, then the Kronecker product 27 Curiously, the hard instances for the Toeplitz construction are very similar to the hard instances for Feature Hashing used in [FKL18]. 28 Due to the choice of matrix names in [BK17].…”