On Using Toeplitz and Circulant Matrices for Johnson–Lindenstrauss Transforms

Freksen, Casper Benjamin; Larsen, Kasper Green

doi:10.1007/s00453-019-00644-y

Cited by 4 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While it might seem at a glance that bounding the high-order moments of X(x) is merely a technical issue, known tools and techniques could not be used to prove Lemmas 3, 4. Particularly, earlier work by Kane and Nelson [KN14,CJN18] and Freksen and Larsen [FL17] used high-order moments bounds as a step in proving probability tail bounds of random variables. The existing techniques, however, can not be adopted to bound high-order moments of X(x) (see also Section 1.2), and novel approaches were needed.…”

Section: Resultsmentioning

confidence: 99%

Fully Understanding the Hashing Trick

Freksen,

Kamma,

Larsen

2018

Preprint

Self Cite

View full text Add to dashboard Cite

Feature hashing, also known as the hashing trick, introduced by Weinberger et al. (2009), is one of the key techniques used in scaling-up machine learning algorithms. Loosely speaking, feature hashing uses a random sparse projection matrix A : R n → R m (where m n) in order to reduce the dimension of the data from n to m while approximately preserving the Euclidean norm. Every column of A contains exactly one non-zero entry, equals to either −1 or 1.Weinberger et al. showed tail bounds on Ax 2 2 . Specifically they showed that for every ε, δ, if x ∞ / x 2 is sufficiently small, and m is sufficiently large, thenThese bounds were later extended by Dasgupta et al. (2010) and most recently refined by Dahlgaard et al.(2017), however, the true nature of the performance of this key technique, and specifically the correct tradeoff between the pivotal parameters x ∞ / x 2 , m, ε, δ remained an open question.We settle this question by giving tight asymptotic bounds on the exact tradeoff between the central parameters, thus providing a complete understanding of the performance of feature hashing. We complement the asymptotic bound with empirical data, which shows that the constants "hiding" in the asymptotic notation are, in fact, very close to 1, thus further illustrating the tightness of the presented bounds in practice.

show abstract

Section: Resultsmentioning

confidence: 99%

Fully Understanding the Hashing Trick

Freksen,

Kamma,

Larsen

2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Moreover, Assumption 6 is also satisfied by certain structured random matrices that enable efficient computation of the compressive mapping, most notably the Fast Johnson-Lindenstrauss transform (Ailon and Chazelle, 2006), and random matrices that exploit sparse matrix multiplications (Kane and Nelson, 2014). The quest for developing efficient Johnson-Lindenstrauss transforms is currently an active research area; some constructions require a slightly larger target dimension in exchange of greater savings in computation time (Freksen and Larsen, 2020). However, the target dimension of order log(q)ǫ −2 is known to be optimal in that any transform that satisfies Assumption 6 uniformly over any set of q points must have a target dimension of this order (Larsen and Nelson, 2017).…”

Section: Main Upper Boundmentioning

confidence: 99%

Statistical optimality conditions for compressive ensembles

Reeve¹,

Kabán²

2021

Preprint

View full text Add to dashboard Cite

We present a framework for the theoretical analysis of ensembles of low-complexity empirical risk minimisers trained on independent random compressions of high-dimensional data. First we introduce a general distribution-dependent upper-bound on the excess risk, framed in terms of a natural notion of compressibility. This bound is independent of the dimension of the original data representation, and explains the in-built regularisation effect of the compressive approach. We then instantiate this general bound to classification and regression tasks, considering Johnson-Lindenstrauss mappings as the compression scheme. For each of these tasks, our strategy is to develop a tight upper bound on the compressibility function, and by doing so we discover distributional conditions of geometric nature under which the compressive algorithm attains minimax-optimal rates up to at most poly-logarithmic factors. In the case of compressive classification, this is achieved with a mild geometric margin condition along with a flexible moment condition that is significantly more general than the assumption of bounded domain.In the case of regression with strongly convex smooth loss functions we find that compressive regression is capable of exploiting spectral decay with near-optimal guarantees. In addition, a key ingredient for our central upper bound is a high probability uniform upper bound on the integrated deviation of dependent empirical processes, which may be of independent interest.

show abstract

“…This includes e.g. five solutions with O(d ln d) embedding time, but different sub-optimal k = O(ε −2 ln n ln 4 d) [16], k = O(ε −2 ln 3 n) [6], k = O(ε −1 ln 3/2 n ln 3/2 d + ε −2 ln n ln 4 d) [16], k = O(ε −2 ln 2 n) [9,23,8] and k = O(ε −2 ln n ln 2 (ln n) ln 3 d) [12], respectively. The second category is solutions where one assumes that k is significantly smaller than d.…”

Section: Introductionmentioning

confidence: 99%

The Fast Johnson-Lindenstrauss Transform is Even Faster

Fandina¹,

Høgsgaard²,

Larsen³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The seminal Fast Johnson-Lindenstrauss (Fast JL) transform by Ailon and Chazelle (SICOMP'09) embeds a set of n points in d-dimensional Euclidean space into optimal k = O(ε −2 ln n) dimensions, while preserving all pairwise distances to within a factor (1 ± ε). The Fast JL transform supports computing the embedding of a data point in O(d ln d + k ln 2 n) time, where the d ln d term comes from multiplication with a d × d Hadamard matrix and the k ln 2 n term comes from multiplication with a sparse k × d matrix. Despite the Fast JL transform being more than a decade old, it is one of the fastest dimensionality reduction techniques for many tradeoffs between ε, d and n.In this work, we give a surprising new analysis of the Fast JL transform, showing that the k ln 2 n term in the embedding time can be improved to (k ln 2 n)/α for an α = Ω(min{ε −1 ln(1/ε), ln n}). The improvement follows by using an even sparser matrix. We also complement our improved analysis with a lower bound showing that our new analysis is in fact tight.

show abstract

On Using Toeplitz and Circulant Matrices for Johnson–Lindenstrauss Transforms

Cited by 4 publications

References 25 publications

Fully Understanding the Hashing Trick

Fully Understanding the Hashing Trick

Statistical optimality conditions for compressive ensembles

The Fast Johnson-Lindenstrauss Transform is Even Faster

Contact Info

Product

Resources

About