ALL-CQS: Adaptive locality-based lossy compression of quality scores

Fu, Jiabing; Dong, Shoubin

doi:10.1109/bibm.2017.8217675

Cited by 5 publications

(3 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is currently research concerned specifically on quality scores compression. 75,76 Since this article is focused on developing a referential compressor for the read sequences stream, the compression of identifiers and quality scores is performed using third-party software. Considering the compression ratio, running times, and software dependencies as reported in a previous study, 25 we selected QUIP 1.1.8.…”

Section: Packing and Unpackingmentioning

confidence: 99%

Tackling the Challenges of FASTQ Referential Compression

Guerra

Lotero

Aedo

et al. 2019

Bioinform Biol Insights

View full text Add to dashboard Cite

The exponential growth of genomic data has recently motivated the development of compression algorithms to tackle the storage capacity limitations in bioinformatics centers. Referential compressors could theoretically achieve a much higher compression than their non-referential counterparts; however, the latest tools have not been able to harness such potential yet. To reach such goal, an efficient encoding model to represent the differences between the input and the reference is needed. In this article, we introduce a novel approach for referential compression of FASTQ files. The core of our compression scheme consists of a referential compressor based on the combination of local alignments with binary encoding optimized for long reads. Here we present the algorithms and performance tests developed for our reads compression algorithm, named UdeACompress. Our compressor achieved the best results when compressing long reads and competitive compression ratios for shorter reads when compared to the best programs in the state of the art. As an added value, it also showed reasonable execution times and memory consumption, in comparison with similar tools.

show abstract

Section: Packing and Unpackingmentioning

confidence: 99%

Tackling the Challenges of FASTQ Referential Compression

Guerra

Lotero

Aedo

et al. 2019

Bioinform Biol Insights

View full text Add to dashboard Cite

show abstract

“…The first category, of QSD lossy compressors, includes QualComp ( Ochoa et al 2013 ), RQS ( Yu et al 2014 ), P/R-Block ( Cánovas et al 2014 ), Quartz ( Yu et al 2015 ), QVZ ( Malysa et al 2015 ), QVZ2 ( Hernaez et al 2016 ), GeneCodeq ( Greenfield et al 2016 ), CROMqs ( No et al 2020 ), ( Fu and Dong 2017 ), Qscomp (lossy) ( Voges et al 2018 ), Crumble ( Bonfield et al 2019 ), ScaleQC ( Yu and Yang 2020 ), and so forth. Those methods overlook the dependence of downstream applications on the initial QSD usage, making the exploration of lossless compressors continue to be a prominent research direction, especially in long-term backup scenarios where data integrity is essential.…”

Section: Introductionmentioning

confidence: 99%

PQSDC: a parallel lossless compressor for quality scores data via sequences partition and run-length prediction mapping

Sun,

Zheng,

Xie

et al. 2024

Bioinformatics

View full text Add to dashboard Cite

Motivation The quality scores data (QSD) account for 70% in compressed FastQ files obtained from the short and long reads sequencing technologies. Designing effective compressors for QSD that counterbalance compression ratio, time cost, and memory consumption is essential in scenarios such as large-scale genomics data sharing and long-term data backup. This study presents a novel parallel lossless QSD-dedicated compression algorithm named PQSDC, which fulfills the above requirements well. PQSDC is based on two core components: a parallel sequences-partition model designed to reduce peak memory consumption and time cost during compression and decompression processes, as well as a parallel four-level run-length prediction mapping model to enhance compression ratio. Besides, the PQSDC algorithm is also designed to be highly concurrent using multi-core CPU clusters. Results We evaluate PQSDC and 4 state-of-the-art compression algorithms on 27 real-world datasets, including 61.857 billion QSD characters and 632.908 million QSD sequences. (1) For short reads, compared to baselines, the maximum improvement of PQSDC reaches 7.06% in average compression ratio, and 8.01% in weighted average compression ratio. During compression and decompression, the maximum total time savings of PQSDC are 79.96% and 84.56%, respectively; the maximum average memory savings are 68.34% and 77.63%, respectively. (2) For long reads, the maximum improvement of PQSDC reaches 12.51% and 13.42% in average and weighted average compression ratio, respectively. The maximum total time savings during compression and decompression are 53.51% and 72.53%, respectively; the maximum average memory savings are 19.44% and 17.42%, respectively. (3) Furthermore, PQSDC ranks second in compression robustness among the tested algorithms, indicating that it is less affected by the probability distribution of the QSD collections. Overall, our work provides a promising solution for QSD parallel compression, which balances storage cost, time consumption, and memory occupation primely. Availability The proposed PQSDC compressor can be downloaded from https://github.com/fahaihi/PQSDC. Supplementary information Supplementary data are available at Bioinformatics online.

show abstract

“…Most algorithms proposed to compress FASTQ quality scores were lossy compression methods ( Cánovas et al, 2014 ; Malysa et al, 2015 ; Fu and Dong, 2017 ; Suaste and Houghten, 2021 ). The effect of lossy compression of quality scores on variant calling was also studied ( Ochoa et al, 2016 ), and showed that the lossy compression can maintain variant calling performance comparable to that with the original data.…”

Section: Introductionmentioning

confidence: 99%

Efficient sequencing data compression and FPGA acceleration based on a two-step framework

Chen,

Wang

et al. 2023

Front. Genet.

View full text Add to dashboard Cite

With the increasing throughput of modern sequencing instruments, the cost of storing and transmitting sequencing data has also increased dramatically. Although many tools have been developed to compress sequencing data, there is still a need to develop a compressor with a higher compression ratio. We present a two-step framework for compressing sequencing data in this paper. The first step is to repack original data into a binary stream, while the second step is to compress the stream with a LZMA encoder. We develop a new strategy to encode the original file into a LZMA highly compressed stream. In addition an FPGA-accelerated of LZMA was implemented to speedup the second step. As a demonstration, we present repaq as a lossless non-reference compressor of FASTQ format files. We introduced a multifile redundancy elimination method, which is very useful for compressing paired-end sequencing data. According to our test results, the compression ratio of repaq is much higher than other FASTQ compressors. For some deep sequencing data, the compression ratio of repaq can be higher than 25, almost four times of Gzip. The framework presented in this paper can also be applied to develop new tools for compressing other sequencing data. The open-source code of repaq is available at: https://github.com/OpenGene/repaq.

show abstract

ALL-CQS: Adaptive locality-based lossy compression of quality scores

Cited by 5 publications

References 20 publications

Tackling the Challenges of FASTQ Referential Compression

Tackling the Challenges of FASTQ Referential Compression

PQSDC: a parallel lossless compressor for quality scores data via sequences partition and run-length prediction mapping

Efficient sequencing data compression and FPGA acceleration based on a two-step framework

Contact Info

Product

Resources

About