Optimal (Euclidean) Metric Compression

Indyk, Piotr; Wagner, Tal

doi:10.1137/20m1371324

Cited by 4 publications

(6 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The above mentioned "binary" version of the Johnson-Lindenstrauss lemma due to [1] (where the entries of the matrix are all either +1 or −1) is particularly important for the following reason. As discussed in [8], an alternate way to convert sketches obtained by the Johnson-Lindenstrauss lemma to bits is possible if the points have bounded integer coordinates and one uses the binary variant of Johnson-Lindenstrauss lemma. This approach is somewhat incomparable to our setting because of the integrality assumption.…”

Section: Previous Variations On Random Projectionmentioning

confidence: 99%

“…However, this is not the optimal number of bits. It was recently shown in [7,8] that if the points are contained in the unit ball and m is the minimum distance between points, then O ǫ −2 n log n + n log log(1/m) bits suffice. Previous to this result, the best known result was that O ǫ −2 n log n log(1/m) bits suffice [10].…”

Section: Distance Compression Beyond Random Mappingsmentioning

confidence: 99%

“…Random projection compresses the data set "point by point" in the sense that the compression process is applied to each point independently from the others. In contrast, the sketches in [8] and [10] must compress the entire data set simultaneously.…”

Section: Distance Compression Beyond Random Mappingsmentioning

confidence: 99%

See 2 more Smart Citations

Expansion of random 0/1 polytopes

Leroux,

Rademacher

2023

Random Struct Algorithms

View full text Add to dashboard Cite

A conjecture of Milena Mihail and Umesh Vazirani (Proc. 24th Annu. ACM Symp. Theory Comput., ACM, Victoria, BC, 1992, pp. 26–38.) states that the edge expansion of the graph of every polytope is at least one. Any lower bound on the edge expansion gives an upper bound for the mixing time of a random walk on the graph of the polytope. Such random walks are important because they can be used to generate an element from a set of combinatorial objects uniformly at random. A weaker form of the conjecture of Mihail and Vazirani says that the edge expansion of the graph of a polytope in is greater than one over some polynomial function of . This weaker version of the conjecture would suffice for all applications. Our main result is that the edge expansion of the graph of a random polytope in is at least with high probability.

show abstract

Section: Previous Variations On Random Projectionmentioning

confidence: 99%

Section: Distance Compression Beyond Random Mappingsmentioning

confidence: 99%

See 1 more Smart Citation

Expansion of random 0/1 polytopes

Leroux,

Rademacher

2023

Random Struct Algorithms

View full text Add to dashboard Cite

show abstract

“…1 for an example. Even in the theoretical literature on optimal vector compression, such clustering plays a crucial role (Indyk & Wagner, 2022). All these quantization methods share one obvious drawback compared to hashing: the model is only quantized after training, thus memory utilization during training is unaffected.…”

Section: Small Clustered Embedding Tablementioning

confidence: 99%

“…We also don't cover the background of "post training" quantization, but refer to the survey, Gray & Neuhoff (1998). Finally, we keep things reasonably heuristic, but for a deep theoretical understanding of metric compression, we recommend Indyk & Wagner (2022). In this framework we can also express most other approaches to training-time table compression:…”

Section: Background and Related Workmentioning

confidence: 99%

Clustering the Sketch: A Novel Approach to Embedding Table Compression

Tsang¹,

Ahle²

2022

Preprint

View full text Add to dashboard Cite

To work with categorical features, machine learning systems employ embedding tables. These tables can become exceedingly large in modern recommendation systems, necessitating the development of new methods for fitting them in memory, even during training. Some of the most successful methods for table compression are Product-and Residual Vector Quantization (Gray & Neuhoff, 1998). These methods replace table rows with references to k-means clustered "codewords." Unfortunately, this means they must first know the table before compressing it, so they can only save memory during inference, not training. Recent work has used hashing-based approaches to minimize memory usage during training, but the compression obtained is inferior to that obtained by "post-training" quantization. We show that the best of both worlds may be obtained by combining techniques based on hashing and clustering. By first training a hashing-based "sketch", then clustering it, and then training the clustered quantization, our method achieves compression ratios close to those of post-training quantization with the training time memory reductions of hashing-based methods. We show experimentally that our method provides better compression and/or accuracy that previous methods, and we prove that our method always converges to the optimal embedding table for least-squares training.

show abstract