Compression Schemes for Similarity Queries

Abstract

We consider compression of sequences in a database so that similarity queries can be performed efficiently in the compressed domain. The fundamental limits for this problem setting, which characterize the tradeoff between compression rate and reliability of the answers to the queries, have been characterized in past work. However, how to approach these limits in practice has remained largely unexplored.Recently, we proposed a scheme for this task that is based on existing lossy compression algorithms, for the general case where the similarity measure satisfies a triangle inequality. Although it was shown that it achieves the fundamental limits for some cases, it is suboptimal in general. In this paper we propose a new scheme that also uses lossy compression algorithms as a building block, but with a carefully chosen distortion measure that is different than the one defining the similarity between sequences. The new scheme significantly improves the compression rate compared to the previously proposed scheme in many cases. For example, for binary sources and Hamming similarity measure, simulation results show a compression rate close to the fundamental limit, and an improvement over the previously proposed scheme of up to 55% (for the same reliability). The results shed light on the fact that compression for similarity identification is inherently different than classical lossy compression.

Computing Similarity Queries for Correlated Gaussian Sources

“…The TC-scheme is an improved version of the LC-scheme by optimizing jointly the quantization distortion and the expected query codeword distance. The results in [5] show that the compression rate of TC-can achieve the identification rate for the case with binary sources and the Hamming distance.…”

“…Proof. Given the stationary Gaussian source { Xn }, we can decompose the source into vectors X of M successive random variables and describe those vectors with a M th order multivariate Gaussian distribution (5). Then we can apply the KLT transform on the decomposed vectors X = A T M X, where A M is the eigenmatrix of the covariance matrix C M .…”

“…However, the optimal D ID -admissible system is difficult to achieve due to the triangle-inequality constraint that most distortion measures possess. The state-of-the-art practical schemes for the similarity identification problem are the triangle-inequality based TC-and LC-schemes proposed in [5], where the TC- scheme is consistently performs better than the LC-scheme. Therefore, we replace the ideal scheme with the practical TCscheme for each component.…”

“…Since it is common to encounter correlated data in the real world, it is of interest to investigate similarity identification schemes for correlated sources. [5] uses lossy compression as a building block to construct the TC-(Type Covering signatures and triangle-inequality decision rule) scheme and the LC-(Lossy Compression signatures and triangle-inequality decision rule) scheme. The LC-scheme only optimizes the quantization distortion and can be achieved by employing a rate-distortion code on the triangle-inequality principle.…”

The Minimal Compression Rate for Similarity Identification

“…the TC-△ and the LC-△ schemes are optimal. However, if X and Y are not equiprobable (and the distortion measure is still Hamming), the LC-△ scheme differs from the TC-△ scheme (see [35,Fig. 2]).…”