Using bit recycling to reduce the redundancy in plurally parsable dictionaries

Al-Rababa'a, Ahmad; Dube, Danny

doi:10.1109/cwit.2015.7255153

Cited by 5 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By having overlapping codewords, we have several chapters, and we can specialize each chapter towards a specific state. Unlike previous multi-dictionary approaches [13,14,15,16], we can not match perfectly one chapter to each state. This is because all chapters must have the same number of words, but the number of words that end in any particular state is variable.…”

Section: Specialized Chaptersmentioning

confidence: 97%

See 1 more Smart Citation

Improving Marlin's Compression Ratio with Partially Overlapping Codewords

Martínez

Sandfort²,

Dube

et al. 2018

2018 Data Compression Conference

Self Cite

View full text Add to dashboard Cite

Marlin [1] is a Variable-to-Fixed (VF) codec optimized for decoding speed. To achieve its speed, Marlin does not encode the current state of the input source, penalyzing compression ratio. In this paper we address this penalty by partially encoding the current state of the input in the lower bits of the codeword. Those bits select which chapter in the dictionary must be used to decode the next codeword. Each chapter is specialized for a subset of states, improving compression ratio. At the same time, we use one victim chapter to encode all rare symbols, increasing the efficiency of the rest of them. The decoding algorithm remains the same, only now codewords have overlapping bits. Mapping techniques allow us to combine common chapters and thus keep an efficient use of the L1 cache. We evaluate our approach with both synthetic and real data sets, and show significant improvements in low entropy sources, where compression efficiency can improve from 93.9% to 98.6%.

show abstract

Section: Specialized Chaptersmentioning

confidence: 97%

“…For one, Tunstall [12] proposed VF dictionaries can be pipelined, but the dictionary must be large to be efficient. Opposed to this, plurally parsable dictionaries [13,14,15,16] are compact, but have temporal dependencies and thus can not be pipelined.…”

Section: Introductionmentioning

confidence: 99%

Improving Marlin's Compression Ratio with Partially Overlapping Codewords

Martínez

Sandfort²,

Dube

et al. 2018

2018 Data Compression Conference

Self Cite

View full text Add to dashboard Cite

show abstract

“…Plurally -i.e., not uniquely-parsable dictionaries [18] have been shown to outperform Tunstall dictionaries of equal size [19,20,21]. We suggest a novel way of constructing plurally parsable dictionaries for arbitrary alphabets using a constrained Markov process of only 255 states (for 8-bit sources).…”

Section: Marlinmentioning

confidence: 98%

Marlin: A High Throughput Variable-to-Fixed Codec Using Plurally Parsable Dictionaries

Martínez

Haurilet

Stiefelhagen

et al. 2017

2017 Data Compression Conference (DCC)

View full text Add to dashboard Cite

We present Marlin, a variable-to-fixed (VF) codec optimized for decoding speed. Marlin builds upon a novel way of constructing VF dictionaries that maximizes efficiency for a given dictionary size. On a lossless image coding experiment, Marlin achieves a compression ratio of 1.94 at 2494MiB/s. Marlin is as fast as state-of-the-art high-throughput codecs (e.g., Snappy, 1.24 at 2643MiB/s), and its compression ratio is close to the best entropy codecs (e.g., FiniteStateEntropy, 2.06 at 523MiB/s). Therefore, Marlin enables efficient and highthroughput encoding for memoryless sources, which was not possible until now.

show abstract

“…Yamamoto and Yokoo described the class of plurally parsable codes called almost-instantaneous variableto-fixed (AIVF) codes, which can also be employed in the non-binary case [47]. Some improvements have been proposed to [47], reporting small gains in terms of compressed sizes [49], [51] and memory efficiency [52]. This work contributes with a new method for constructing VF codes that provide an advantageous trade-off between compression efficiency and computational complexity, specifically for the decoder.…”

Section: Related Workmentioning

confidence: 99%

“…Using tree notation, words associated with non-leaf nodes can also be included in W when removing the prefix-free constraint. This has enabled more flexible and efficient designs [47], [49]- [51], [53].…”

Section: Plurally Parsable Dictionariesmentioning

confidence: 99%

High-Throughput Variable-to-Fixed Entropy Codec Using Selective, Stochastic Code Forests

et al. 2020

View full text Add to dashboard Cite

Efficient high-throughput (HT) compression algorithms are paramount to meet the stringent constraints of present and upcoming data storage, processing, and transmission systems. In particular, latency, bandwidth and energy requirements are critical for those systems. Most HT codecs are designed to maximize compression speed, and secondarily to minimize compressed lengths. On the other hand, decompression speed is often equally or more critical than compression speed, especially in scenarios where decompression is performed multiple times and/or at critical parts of a system. In this work, an algorithm to design variable-to-fixed (VF) codes is proposed that prioritizes decompression speed. Stationary Markov analysis is employed to generate multiple, jointly optimized codes (denoted code forests). Their average compression efficiency is on par with the state of the art in VF codes, e.g., within 1% of Yamamoto et al.'s algorithm. The proposed code forest structure enables the implementation of highly efficient codecs, with decompression speeds 3.8 times faster than other state-of-the-art HT entropy codecs with equal or better compression ratios for natural data sources. Compared to these HT codecs, the proposed forests yields similar compression efficiency and speeds. INDEX TERMS Data compression, high-throughput entropy coding, variable-to-fixed codes.

show abstract

Using bit recycling to reduce the redundancy in plurally parsable dictionaries

Cited by 5 publications

References 7 publications

Improving Marlin's Compression Ratio with Partially Overlapping Codewords

Improving Marlin's Compression Ratio with Partially Overlapping Codewords

Marlin: A High Throughput Variable-to-Fixed Codec Using Plurally Parsable Dictionaries

High-Throughput Variable-to-Fixed Entropy Codec Using Selective, Stochastic Code Forests

Contact Info

Product

Resources

About