Improving Marlin's Compression Ratio with Partially Overlapping Codewords

Abstract

Marlin [1] is a Variable-to-Fixed (VF) codec optimized for decoding speed. To achieve its speed, Marlin does not encode the current state of the input source, penalyzing compression ratio. In this paper we address this penalty by partially encoding the current state of the input in the lower bits of the codeword. Those bits select which chapter in the dictionary must be used to decode the next codeword. Each chapter is specialized for a subset of states, improving compression ratio. At the same time, we use one victim chapter to encode all rare symbols, increasing the efficiency of the rest of them. The decoding algorithm remains the same, only now codewords have overlapping bits. Mapping techniques allow us to combine common chapters and thus keep an efficient use of the L1 cache. We evaluate our approach with both synthetic and real data sets, and show significant improvements in low entropy sources, where compression efficiency can improve from 93.9% to 98.6%.

Compression and Pattern Matching

“…In recent years, Martinez et al [31] proposed a novel compression method called Marlin and an improvement of it [32]. These methods achieve both decompression at ultrahigh-speed good performance in terms of compression ratio.…”

Rice-Marlin Codes: Tiny and Efficient Variable-to-Fixed Codes

“…Most VF codecs do uncompress a message by consuming N bits from the compressed stream and emit its corresponding word. Marlin uses instead an overlapping codeword methodology [2], where N bits are peeked from the stream, corresponding to one entry in the dictionary, but only K bits are consumed, as seen in Fig. 1.…”

“…Marlin [1,2] was the first VF codec that achieved a competitive compression ratio at HT decoding speeds thanks to the combination of using plurally parsable dictionaries [21,22,23,24,25] in a memoryless context, however, being a VF codec, Marlin is slow to encode. The main bottleneck during the encoding process is related to the size of the dictionary.…”

“…

Marlin [1,2] is a Variable-to-Fixed (VF) codec optimized for high decoding speed through the use of small sized dictionaries that fit in the L1 cache of most CPUs. While the size of Marlin dictionaries is adequate for decoding, they are still too large to be encoded fast.

…”

High-Throughput Variable-to-Fixed Entropy Codec Using Selective, Stochastic Code Forests

“…This enables the design of more efficient codes and minimizes the impact of low-probability symbols and of incompressible noise. The proposed codec extends upon our previous conference works [21], [54] with theoretical contributions, exhaustive experimental results to assess their performance, and a standalone code container released with digital object identifier (DOI) 10.24433/CO.2752092.v2 1 as supplementary materials to enable full experimental reproducibility.…”