Lossless Compression of Time Series Data with Generalized Deduplication

Vestergaard, Rasmus; Zhang, Qi; Lucani, Daniel E.

doi:10.1109/globecom38437.2019.9013957

Cited by 25 publications

(13 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In "Lossless Compression of Time Series Data with Generalized Deduplication" [20], the authors presented a new strategy for data deduplication to provide compressed data storage for large amounts of time series data. In their approach, they split each data chunk into a part that must be stored directly and a part worth deduplicating.…”

Section: Introductionmentioning

confidence: 99%

The Use of Template Miners and Encryption in Log Message Compression

2021

View full text Add to dashboard Cite

Presently, almost every computer software produces many log messages based on events and activities during the usage of the software. These files contain valuable runtime information that can be used in a variety of applications such as anomaly detection, error prediction, template mining, and so on. Usually, the generated log messages are raw, which means they have an unstructured format. This indicates that these messages have to be parsed before data mining models can be applied. After parsing, template miners can be applied on the data to retrieve the events occurring in the log file. These events are made from two parts, the template, which is the fixed part and is the same for all instances of the same event type, and the parameter part, which varies for all the instances. To decrease the size of the log messages, we use the mined templates to build a dictionary for the events, and only store the dictionary, the event ID, and the parameter list. We use six template miners to acquire the templates namely IPLoM, LenMa, LogMine, Spell, Drain, and MoLFI. In this paper, we evaluate the compression capacity of our dictionary method with the use of these algorithms. Since parameters could be sensitive information, we also encrypt the files after compression and measure the changes in file size. We also examine the speed of the log miner algorithms. Based on our experiments, LenMa has the best compression rate with an average of 67.4%; however, because of its high runtime, we would suggest the combination of our dictionary method with IPLoM and FFX, since it is the fastest of all methods, and it has a 57.7% compression rate.

show abstract

Section: Introductionmentioning

confidence: 99%

The Use of Template Miners and Encryption in Log Message Compression

2021

View full text Add to dashboard Cite

show abstract

“…Although originally considered for large scale data storage [21,32], GD has been adapted to multi-source data compression protocols [15] and file compression for time-series data [35,37]. This has resulted in lightweight, online compression mechanisms suitable to the Internet of Things (IoT) an file compressors with excellent random access properties.…”

Section: Introductionmentioning

confidence: 99%

ZipLine

Vaucher

Yazdani

Felber

et al. 2020

Proceedings of the 16th International Conference on Emerging Networking EXperiments and Technologies

Self Cite

View full text Add to dashboard Cite

Network appliances continue to offer novel opportunities to offload processing from computing nodes directly into the data plane. One popular concern of network operators and their customers is to move data increasingly faster. A common technique to increase data throughput is to compress it before its transmission. However, this requires compression of the data-a time and energy demanding preprocessing phase-and decompression upon reception-a similarly resource consuming operation. Moreover, if multiple nodes transfer similar data chunks across the network hop (e.g., a given pair of switches), each node effectively wastes resources by executing similar steps. This paper proposes ZipLine, an approach to design and implement (de)compression at line speed leveraging the Tofino hardware platform which is programmable using the P4 16 language. We report on lessons learned while building the system and show throughput, latency and compression measurements on synthetic and real-world traces, showcasing the benefits and trade-offs of our design.

show abstract

“…The method has practical merits, and has been shown to achieve a compression of modelled sensor data in many cases where deduplication is unable to [5]. Another instance is able to achieve a compression comparable to typical lossless compression methods for ECG data, while maintaining benefits from classic deduplication [6]. This paper is a study of the theoretical properties of the technique, and it is shown how generalized deduplication compares to classic deduplication.…”

Section: Introductionmentioning

confidence: 99%

Generalized Deduplication: Bounds, Convergence, and Asymptotic Properties

Vestergaard

Lucani

2019

2019 IEEE Global Communications Conference (GLOBECOM)

Self Cite

View full text Add to dashboard Cite

We study a generalization of deduplication, which enables lossless deduplication of highly similar data and show that classic deduplication with fixed chunk length is a special case. We provide bounds on the expected length of coded sequences for generalized deduplication and show that the coding has asymptotic near-entropy cost under the proposed source model. More importantly, we show that generalized deduplication allows for multiple orders of magnitude faster convergence than classic deduplication. This means that generalized deduplication can provide compression benefits much earlier than classic deduplication, which is key in practical systems. Numerical examples demonstrate our results, showing that our lower bounds are achievable, and illustrating the potential gain of using the generalization over classic deduplication. In fact, we show that even for a simple case of generalized deduplication, the gain in convergence speed is linear with the size of the data chunks.

show abstract

Lossless Compression of Time Series Data with Generalized Deduplication

Cited by 25 publications

References 14 publications

The Use of Template Miners and Encryption in Log Message Compression

The Use of Template Miners and Encryption in Log Message Compression

ZipLine

Generalized Deduplication: Bounds, Convergence, and Asymptotic Properties

Contact Info

Product

Resources

About