Faster Compact Top-k Document Retrieval

Abstract

An optimal index solving top-k document retrieval [Navarro and Nekrich, SODA'12] takes O(m + k) time for a pattern of length m, but its space is at least 80n bytes for a collection of n symbols. We reduce it to 1.5n-3n bytes, with O(m+(k+log log n) log log n) time, on typical texts. The index is up to 25 times faster than the best previous compressed solutions, and requires at most 5% more space in practice (and in some cases as little as one half). Apart from replacing classical by compressed data structures, our main idea is to replace suffix tree sampling by frequency thresholding to achieve compression.

Efficient Compressed Indexing for Approximate Top-k String Retrieval

“…We use various document collections, following previous work [13,7] and exploring different aspects of statistical compressibility, size, number of documents, and repetitiveness: ClueWiki (English, few large documents), DNA (synthetic, mildly repetitive with 5% mutations among documents), KGS (Go game records), Wiki (more and shorter documents), Proteins (many more documents, almost incompressible), and TodoCL (a snapshot of the Chilean Web, with real queries, used to measure quality). Table 1 shows their main characteristics (column "compress" shows how the LZ78-based Unix Compress program compresses them).…”

“…Navarro and Nekrich [12] reduced the time to the optimal O(m + k). Konow and Navarro [7] implemented this index, obtaining an index that uses 20-30 bits per character (bpc) 1 and answers top-k queries in k to 4k microseconds (µsec). Their time complexity is O(m + (k + log log n) log log n) with high probability, on statistically typical texts [15].…”

New space/time tradeoffs for top- k document retrieval on sequences

“…Konow and Navarro [21] achieved O(p + (lg lg n) 2 + k lg lg n) time within |CSA| + (n lg D + 4n lg lg n)(1 + o(1)) bits, but the result holds only almost surely on typical texts, not in the worst case. Their index, on the other hand, turns out to be very competitive in practice.…”

“…For the first part, we can use a data structure [21], which is built in O(n lg n) time and answers a top-k query in O((k+ lg n) lg n) time once the locus of the pattern is given. 4 Adding the O(k lg n) times over all the O(n/c) sampled nodes, for c = k lg 2 k lg ε n, gives O(n lg 1−ε n/ lg 2 k), which added over all the powers of 2 for k gives O(n lg 1−ε n).…”

“…The structure for the case lg σ < lg n can be built in time O(σ lg lg n lg lg n lg 2 n) = o(n), using the data structure [21] analyzed in Section 3.3. Therefore the construction time O(n lg 2 n lg * n) of the main structure dominates.…”