Compression: a key for next-generation text retrieval systems

String Processing and Information Retrieval

et al. 2003

Abstract. This work presents (s, c)-Dense Code, a new method for compressing natural language texts. This technique is a generalization of a previous compression technique called End-Tagged Dense Code that obtains better compression ratio as well as a simpler and faster encoding than Tagged Huffman. At the same time, (s, c)-Dense Code is a prefix code that maintains the most interesting features of Tagged Huffman Code with respect to direct search on the compressed text. (s, c)-Dense Coding retains all the efficiency and simplicity of Tagged Huffman, and improves its compression ratios. We formally describe the (s, c)-Dense Code and show how to compute the parameters s and c that optimize the compression for a specific corpus. Our empirical results show that (s, c)-Dense Code improves End-Tagged Dense Code and Tagged Huffman Code, and reaches only 0.5% overhead over plain Huffman Code.

Section: Tagged Huffman Codesmentioning

confidence: 99%

“…Then the obtained codeword is searched for in the compressed text using any classical string matching algorithm with no modifications. They call this technique direct searching [12,17].…”

Section: Tagged Huffman Codesmentioning

confidence: 99%

Section: Word-based Huffman Compressionmentioning

confidence: 99%

Section: Word-based Huffman Compressionmentioning

confidence: 99%

See 2 more Smart Citations

(S,C)-Dense Coding: An Optimized Compression Code for Natural Language Text Databases

Brisaboa

Fariña

String Processing and Information Retrieval

et al. 2003

“…A popular approach is to parse the source text into syntactic units, such as q-grams, syllables, or words, and run a compressor tailored to handle a sequence of unit identifiers [12,11,32,33,34,15,35,36,20].…”

Section: Preprocessing For Compressionmentioning

confidence: 99%

Boosting Text Compression with Word-Based Statistical Encoding

Fariña

Paramá

2011

The Computer Journal

Self Cite

Semistatic word-based byte-oriented compressors are known to be attractive alternatives to compress natural language texts. With compression ratios around 30-35%, they allow fast direct searching of compressed text. In this article we reveal that these compressors have even more benefits. We show that most of the state-of-the-art compressors benefit from compressing not the original text, but the compressed representation obtained by a word-based byte-oriented statistical compressor. For example, p7zip with a dense-coding preprocessing achieves even better compression ratios and much faster compression than p7zip alone. We reach compression ratios below 17% in typical large English texts, which was obtained only by the slow PPM compressors. Furthermore, searches perform much faster if the final compressor operates over word-based compressed text. We show that typical self-indexes also profit from our preprocessing step. They achieve much better space and time performance when indexing is preceded by a compression step. Apart from using the well-known Tagged Huffman code, we present a new suffix-free Dense-Code-based compressor that compresses slightly better. We also show how some self-indexes can handle non-suffix-free codes. As a result, the compressed/indexed text requires around 35% of the space of the original text and allows indexed searches for both words and phrases. * A preliminary partial version of this work appeared in [1].

Lempel‐Ziv compression of highly structured documents

Adiego

Fuente

2007

J. Am. Soc. Inf. Sci.

The authors describe Lempel-Ziv to Compress Structure (LZCS), a novel Lempel-Ziv approach suitable for compressing structured documents. LZCS takes advantage of repeated substructures that may appear in the documents, by replacing them with a backward reference to their previous occurrence. The result of the LZCS transformation is still a valid structured document, which is human-readable and can be transmitted by ASCII channels. Moreover, LZCS transformed documents are easy to search, display, access at random, and navigate. In a second stage, the transformed documents can be further compressed using any semistatic technique, so that it is still possible to do all those operations efficiently; or with any adaptive technique to boost compression. LZCS is especially efficient in the compression of collections of highly structured data, such as extensible markup language (XML) forms, invoices, e-commerce, and Web-service exchange documents. The comparison with other structure-aware and standard compressors shows that LZCS is a competitive choice for these type of documents, whereas the others are not well-suited to support navigation or random access. When joined to an adaptive compressor, LZCS obtains by far the best compression ratios.