LFZip: Lossy Compression of Multivariate Floating-Point Time Series Data via Improved Prediction

Chandak, Shubham; Tatwawadi, Kedar; Wen, Chengtao; Wang, Lingyun; Ojea, Juan Aparicio; Weissman, Tsachy

doi:10.1109/dcc47342.2020.00042

Cited by 29 publications

(25 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Both SZ and LFZip can compress millions of samples per second, with SZ being about an order of magnitude faster than LFZip (Chandak et al, 2020). Since LFZip simply performs uniform scalar quantization and entropy coding (in the mode used here), we believe that it can be significantly optimized further for this application.…”

Section: Resultsmentioning

confidence: 99%

“…This gives rise to a tradeoff between the compressed size and the distortion, referred to as the rate-distortion curve. Here, we work with two state-of-the-art lossy compressors for time-series data, LFZip (Chandak et al, 2020) and SZ (Di and Cappello, 2016;Tao et al, 2017;Liang et al, 2018). Both these compressors work with a maxerror parameter that specifies the maximum absolute deviation between the original and the reconstructed data.…”

Section: Lossy Compressionmentioning

confidence: 99%

“…In this context, lossy compression is a natural candidate to provide a boost in compression at the cost of certain amount of distortion in the raw signal. There have been several works on lossy compression for time series data, including SZ (Di and Cappello, 2016;Tao et al, 2017;Liang et al, 2018) and LFZip (Chandak et al, 2020) that provide a guarantee that the reconstruction lies within a certain user-defined interval of the original value for all time steps. However, in the case of nanopore raw current signal, the metric of interest is not the maximum deviation from the original value, but instead the impact on the performance of basecalling and other downstream analysis steps.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Impact of lossy compression of nanopore raw signal data on basecalling and consensus accuracy

Chandak

Tatwawadi

Sridhar

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Motivation: Nanopore sequencing provides a real-time and portable solution to genomic sequencing, with long reads enabling better assembly and structural variant discovery than second generation technologies. The nanopore sequencing process generates huge amounts of data in the form of raw current data, which must be compressed to enable efficient storage and transfer. Since the raw current data is inherently noisy, lossy compression has potential to significantly reduce space requirements without adversely impacting performance of downstream applications. Results: We study the impact of two state-of-the-art lossy time-series compressors applied to nanopore raw current data, evaluating the tradeoff between compressed size and basecalling/consensus accuracy. We test several basecallers and consensus tools on a variety of datasets at varying depths of coverage, and conclude that lossy compression can provide as much as 40-50% reduction in compressed size of raw signal data with negligible impact on basecalling and consensus accuracy. Availability: The code, data and supplementary results are available at https://github.com/ shubhamchandak94/lossy_compression_evaluation.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Lossy Compressionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Impact of lossy compression of nanopore raw signal data on basecalling and consensus accuracy

Chandak

Tatwawadi

Sridhar

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, GPU hardware for IoT Edge devices is not yet mainstream. On the other hand, in [5], the removal of noise is identified as an opportunity for higher compression rates on IoT devices. An important consideration is that preprocessing and lossy approaches traditionally have a trade off resulting in sacrificing both specificity and sensitivity in the resulting data.…”

Section: Immediate Processing and Actuation Without Transmissionmentioning

confidence: 99%

Vibration Edge Computing in Maritime IoT

Michala

Vourganas

Coraddu

2021

ACM Trans. Internet Things

View full text Add to dashboard Cite

IoT and the Cloud are among the most disruptive changes in the way we use data today. These changes have not significantly influenced practices in condition monitoring for shipping. This is partly due to the cost of continuous data transmission. Several vessels are already equipped with a network of sensors. However, continuous monitoring is often not utilised and onshore visibility is obscured. Edge computing is a promising solution but there is a challenge sustaining the required accuracy for predictive maintenance. We investigate the use of IoT systems and Edge computing, evaluating the impact of the proposed solution on the decision making process. Data from a sensor and the NASA-IMS open repository were used to show the effectiveness of the proposed system and to evaluate it in a realistic maritime application. The results demonstrate our real-time dynamic intelligent reduction of transmitted data volume by without sacrificing specificity or sensitivity in decision making. The output of the Decision Support System fully corresponds to the monitored system's actual operating condition and the output when the raw data are used instead. The results demonstrate that the proposed more efficient approach is just as effective for the decision making process.

show abstract

“…Moura e Hutchison (2020) mencionaram que os dispositivos de edge computing podem "sintetizar dados brutos" para reduzir o volume de dados na IoT. Alguns trabalhos avaliaram técnicas de redução de dados em sistemas IoT [Spiegel et al, 2018], [Routray et al, 2020], [Gia et al, 2019], [Chandak et al, 2020], [Blalock et al, 2018]. Porém, essas investigações não avaliaram um cenário de desconexão entre os estágios de IoT, nem consideraram a compressão dos dados cifrados na bruma computacional.…”

Section: Trabalhos Relacionadosunclassified

Resiliência de Dados da Bruma Computacional na Internet das Coisas

Ribeiro

Kamienski

2021

Anais Do XXXIX Simpósio Brasileiro De Redes De Computadores E Sistemas Distribuídos (SBRC 2021)

View full text Add to dashboard Cite

Um sistema IoT baseado em bruma (mist) e névoa computacional deve resistir a desconexões e a interrupções da névoa, já que os dados transmitidos pela bruma podem ser perdidos. Esse artigo propõe uma solução intitulada ReMITS, que persiste os dados na bruma, mesmo durante desconexões de rede. Também foi proposto um algoritmo de compressão (LoRa-SAX), para reduzir o atraso dos dados quando a conexão é retomada. O ReMITS foi avaliado com uma carga simulada de 5.000 sensores e com sete configurações, para 1, 5 e 30 minutos de desconexão. Foi observado que o ReMITS entregou todos os pacotes à névoa e que o LoRa-SAX combinado ao algoritmo bzip2 reduziu o tempo de chegada dos pacotes em até 98,5% e o tamanho dos dados em até 93,6%.

show abstract

LFZip: Lossy Compression of Multivariate Floating-Point Time Series Data via Improved Prediction

Cited by 29 publications

References 16 publications

Impact of lossy compression of nanopore raw signal data on basecalling and consensus accuracy

Impact of lossy compression of nanopore raw signal data on basecalling and consensus accuracy

Vibration Edge Computing in Maritime IoT

Resiliência de Dados da Bruma Computacional na Internet das Coisas

Contact Info

Product

Resources

About