LIPT: a lossless text transform to improve compression

Awan, Fauzia; Mukherjee, Amar

doi:10.1109/itcc.2001.918838

Cited by 35 publications

(28 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here are major methods available for text preprocessing algorithm with static dictionary like Star Encoding, LIPT [2], StarNT. Skibinski and Grabowski [17] suggested word based replacement transformation method with static dictionary and introduced the recognized preprocessing methods like EOL coding, Capital Conversion, Q-gram replacement, Binary data Filter and Surrounding words with spaces.…”

Section: Methodsmentioning

confidence: 99%

“…The main areas of lossless text compression are generation of new compressors or transformation algorithm. Researchers have proved that word based preprocessing method saves the run time memory, boost the compression rates and also speeds up the transmission time [12,15].Though there are methods which have been existing based on word replacement preprocessing techniques, like Star Encoding [6], LIPT [2] and StarNT [18], it is known that there can be a better word based preprocessing techniques are possible as the days and technologies advancement. The most common preprocessing algorithm used in practice is the dictionary based scheme.…”

Section: Introductionmentioning

confidence: 99%

“…The dictionary can be generated as either static or dynamic manner. There are major methods available for text preprocessing algorithm with static dictionary like Star Encoding, Length-Preserving Transform (LPT), Reverse Length-Preserving Transform (RLPT), Shortened -Context Length-Preserving Transform (SCLPT) Length Index Preserving Transform LIPT [2] and StarNT [18]. The main drawback of the algorithms is that of a fixed initial storage overhead of 1 MB in the form of shared dictionaries [6].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Semi-Adaptive Substitution Coder for Lossless Text Compression

Rexline¹,

Robert²,

Lobo³

2013

IJCA

View full text Add to dashboard Cite

In this paper, a new text transformation technique called Semi-Adaptive Substitution Coder for Lossless TextCompression is proposed. The rapid advantage of this Substitution Coder is that it substitutes the codewords by referring the reference of the word's position in the dictionary to expedite the dictionary mapping and also codewords are shorter than words and, thus, the same amount of text will require less space. In general, text transformation needs an external dictionary to store the frequently used words. To preserve this transformation method in a healthy way, a semiadaptive dictionary is used and therefore which reduces the expenditure of memory overhead and speeds up the transformation because of the smaller size dictionary. This new transformation algorithm is implemented and tested using Calgary Corpus and Large Corpus. In this implementation Semi-Adaptive Substitution Coder in connection with a popular bzip2 and commonly used Gzip compressors improve the compression performance by about 7-9% on large files.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semi-Adaptive Substitution Coder for Lossless Text Compression

Rexline¹,

Robert²,

Lobo³

2013

IJCA

View full text Add to dashboard Cite

show abstract

“…Research work in star-based encoding techniques [1,7,11,15], dictionary-based encoding techniques [12,13,14] and digram encoding techniques [5,9,17] are intended for text files. Another category is of data transformation techniques like DCT and wavelet used on image files.…”

Section: Literature Reviewmentioning

confidence: 99%

Achieving Better Compression Applying Index-based Byte-Pair Transformation before Arithmetic Coding

Doshi¹,

Gandhi²

2014

IJCA

View full text Add to dashboard Cite

show abstract

“…Alternatively, there are various ad-hoc heuristics [37,38,39,40,41,42] that enrich a general-purpose compressor by a preprocessing that chooses some words or other types of syntactic units and replaces them by different kinds of identifiers, usually assigning shorter identifiers to more frequent units. Those are coupled with a number of very specific tricks that slightly improve the compression ratio.…”

Section: Preprocessing For Compressionmentioning

confidence: 99%

Boosting Text Compression with Word-Based Statistical Encoding

Fariña

Navarro

Paramá

2011

The Computer Journal

View full text Add to dashboard Cite

Semistatic word-based byte-oriented compressors are known to be attractive alternatives to compress natural language texts. With compression ratios around 30-35%, they allow fast direct searching of compressed text. In this article we reveal that these compressors have even more benefits. We show that most of the state-of-the-art compressors benefit from compressing not the original text, but the compressed representation obtained by a word-based byte-oriented statistical compressor. For example, p7zip with a dense-coding preprocessing achieves even better compression ratios and much faster compression than p7zip alone. We reach compression ratios below 17% in typical large English texts, which was obtained only by the slow PPM compressors. Furthermore, searches perform much faster if the final compressor operates over word-based compressed text. We show that typical self-indexes also profit from our preprocessing step. They achieve much better space and time performance when indexing is preceded by a compression step. Apart from using the well-known Tagged Huffman code, we present a new suffix-free Dense-Code-based compressor that compresses slightly better. We also show how some self-indexes can handle non-suffix-free codes. As a result, the compressed/indexed text requires around 35% of the space of the original text and allows indexed searches for both words and phrases. * A preliminary partial version of this work appeared in [1].

show abstract

LIPT: a lossless text transform to improve compression

Abstract: We propose an approach to develop

Cited by 35 publications

References 15 publications

Semi-Adaptive Substitution Coder for Lossless Text Compression

Semi-Adaptive Substitution Coder for Lossless Text Compression

Achieving Better Compression Applying Index-based Byte-Pair Transformation before Arithmetic Coding

Boosting Text Compression with Word-Based Statistical Encoding

Contact Info

Product

Resources

About