Marlin: A High Throughput Variable-to-Fixed Codec Using Plurally Parsable Dictionaries

Abstract

We present Marlin, a variable-to-fixed (VF) codec optimized for decoding speed. Marlin builds upon a novel way of constructing VF dictionaries that maximizes efficiency for a given dictionary size. On a lossless image coding experiment, Marlin achieves a compression ratio of 1.94 at 2494MiB/s. Marlin is as fast as state-of-the-art high-throughput codecs (e.g., Snappy, 1.24 at 2643MiB/s), and its compression ratio is close to the best entropy codecs (e.g., FiniteStateEntropy, 2.06 at 523MiB/s). Therefore, Marlin enables efficient and highthroughput encoding for memoryless sources, which was not possible until now.

Improving Marlin's Compression Ratio with Partially Overlapping Codewords

“…To generate the chapters, we extend the algorithm used to create dictionaries in [1]. We define A = {a 1 , a 2 , • • • , a N } as an alphabet of input symbols sorted in order of non-increasing probability (i.e., P (a n ) ≥ P (a n+1 ), ∀n).…”

“…Given a dictionary, we can calculate the steady probability of each state using a Markov model, as described in [1], but due to the fast convergence rate of the process this is unnecessary. We simply estimateP l (w) for each w in W using Eqn.…”

“…In 2017, Marlin [1] was introduced. Marlin is a VF codec based on plurally parsable dictionaries that achieves compelling compression ratios while being faster than most HT codecs.…”

“…1, hence we name this technique Partially Overlapping Codewords. The decoding algorithm which we used in [1] can be straightforwardly modified to cope with the overlap without any speed penalty.…”

“…Here, O represents the number of overlapping bits between codewords. For the case O = 0, the format is identical to that of the original Marlin [1].…”