Index compression through document reordering

Blandford, Daniel K.; Blelloch, Guy E.

doi:10.1109/dcc.2002.999972

Cited by 73 publications

(89 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…5 We can represent f using nH 0 (f ) + o(n)(H 0 (f ) + 1) bits so that any f (i) can be computed in O (1) time. Moreover, each element of f −1 (a) can be computed in O (lg lg σ) time, and |f −1 (a)| requires time O (lg lg σ lg lg lg σ).…”

Section: Compressing Functions and Dynamic Collections Of Disjoint Setsmentioning

confidence: 99%

Alphabet Partitioning for Compressed Rank/Select and Applications

Barbay

Gagie

Navarro

et al. 2010

Algorithms and Computation

View full text Add to dashboard Cite

Abstract. We present a data structure that stores a string s[1..n] over the alphabet [1..σ] in nH0(s) + o(n)(H0(s)+1) bits, where H0(s) is the zero-order entropy of s. This data structure supports the queries access and rank in time O (lg lg σ), and the select query in constant time. This result improves on previously known data structures using nH0(s) + o(n lg σ) bits, where on highly compressible instances the redundancy o(n lg σ) cease to be negligible compared to the nH0(s) bits that encode the data. The technique is based on combining previous results through an ingenious partitioning of the alphabet, and practical enough to be implementable. It applies not only to strings, but also to several other compact data structures. For example, we achieve (i) faster search times and lower redundancy for the smallest existing full-text self-index; (ii) compressed permutations π with times for π() and π −1 () improved to log-logarithmic; and (iii) the first compressed representation of dynamic collections of disjoint sets.

show abstract

Section: Compressing Functions and Dynamic Collections Of Disjoint Setsmentioning

confidence: 99%

Alphabet Partitioning for Compressed Rank/Select and Applications

Barbay

Gagie

Navarro

et al. 2010

Algorithms and Computation

View full text Add to dashboard Cite

show abstract

“…Beyond social sciences, data ordering has been popular in gene expression data analysis in bioinformatics [13], [14] and analysis of geographical data [34]. It is also important in bandwidth minimization [35] and data compression [36], [37].…”

Section: B Data Reorderingmentioning

confidence: 99%

Efficient Visualization of Large-Scale Data Tables through Reordering and Entropy Minimization

Djuric

Vučetić

2013

2013 IEEE 13th International Conference on Data Mining

View full text Add to dashboard Cite

Abstract-Visualization of data tables with n examples and m columns using heatmaps provides a holistic view of the original data. As there are n! ways to order rows and m! ways to order columns, and data tables are typically ordered without regard to visual inspection, heatmaps of the original data tables often appear as noisy images. However, if rows and columns of a data table are ordered such that similar rows and similar columns are grouped together, a heatmap may provide a deep insight into the underlying data distribution. We propose an informationtheoretic approach to produce a well-ordered data table. In particular, we search for ordering that minimizes entropy of residuals of predictive coding applied on the ordered data table. This formalization leads to a novel ordering procedure, EMordering, that can be applied separately on rows and columns. For ordering of rows, EM-ordering repeats until convergence the steps of (1) rescaling columns and (2) solving a Traveling Salesman Problem (TSP) where rows are treated as cities. To allow fast ordering of large data tables, we propose an efficient TSP heuristic with modest O n log(n) time complexity. When compared to the existing state-of-the-art reordering approaches, we show that the method often provides heatmaps of higher visual quality, while being significantly more scalable. Moreover, analysis of real-world traffic and financial data sets using the proposed method, which allowed us to readily gain deeper insights about the data, further confirmed that EM-ordering can be a valuable tool for visual exploration of large-scale data sets.

show abstract

“…Shieh et al [9] and Blandford and Blelloch [2] built a full document similarity graph and traversed it by different algorithms such as the TSP and recursive splitting. Silvestri et al [10] used an on the fly assignment technique with temporal and spacial complexity linear or superlinear on the number of documents, but also dependant on the average document length.…”

Section: Implementation Considerationsmentioning

confidence: 99%

“…This process is done after the collection is traversed and the inverted file is built. These works are presented in [2] and [9], in the following of this paper, the B&B (Blandford and Blelloch) and the TSP approach (Travelling Salesman Problem) respectively. Results show that the document identifier reassignment technique is effective in lowering the average d-gap, and therefore allowing gains in compression ratios.…”

Section: Introductionmentioning

confidence: 99%

Document Identifier Reassignment Through Dimensionality Reduction

Blanco

Barreiro

2005

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Most modern retrieval systems use compressed Inverted Files (IF) for indexing. Recent works demonstrated that it is possible to reduce IF sizes by reassigning the document identifiers of the original collection, as it lowers the average distance between documents related to a single term. Variable-bit encoding schemes can exploit the average gap reduction and decrease the total amount of bits per document pointer. However, approximations developed so far requires great amounts of time or use an uncontrolled memory size. This paper presents an efficient solution to the reassignment problem consisting in reducing the input data dimensionality using a SVD transformation. We tested this approximation with the Greedy-NN TSP algorithm and one more efficient variant based on dividing the original problem in sub-problems. We present experimental tests and performance results in two TREC collections, obtaining good compression ratios with low running times. We also show experimental results about the tradeoff between dimensionality reduction and compression, and time performance.

show abstract

Index compression through document reordering

Cited by 73 publications

References 7 publications

Alphabet Partitioning for Compressed Rank/Select and Applications

Alphabet Partitioning for Compressed Rank/Select and Applications

Efficient Visualization of Large-Scale Data Tables through Reordering and Entropy Minimization

Document Identifier Reassignment Through Dimensionality Reduction

Contact Info

Product

Resources

About