LFQC: a lossless compression algorithm for FASTQ files

Nicolae, Marius; Pathak, Sudipta; Rajasekaran, Sanguthevar

doi:10.1093/bioinformatics/btv384

Cited by 52 publications

(29 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there is still some distance away from practical usage. Once its compression speed becomes acceptable in practical usage, existing excellent quality score compressors (not limited to quality score compression) would become acceptable since many compressors [4,5,10,12] use libzpaq as their backend compressor. Thus, it is of great importance to optimize libzpaq's performance of compression speed.…”

Section: Parallelization Methods For Libzpaq Using Simd Techniquementioning

confidence: 99%

“…However, these four criteria are not independent but mutually restrictive. Most existing lossless quality score compressors [4][5][6] adopt the design pattern of "sacrificing one for another" in the classic evaluating paradigm of "Compression Ratio, Compression Speed, Decompression Speed, and Memory usage". For instance, some compressors try to sacrifice the compression ratio by applying a simple probability model to compress or decompress at a very high speed.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LCQS: an efficient lossless compression tool of quality scores with random access functionality

Dong

2020

BMC Bioinformatics

View full text Add to dashboard Cite

Background: Advanced sequencing machines dramatically speed up the generation of genomic data, which makes the demand of efficient compression of sequencing data extremely urgent and significant. As the most difficult part of the standard sequencing data format FASTQ, compression of the quality score has become a conundrum in the development of FASTQ compression. Existing lossless compressors of quality scores mainly utilize specific patterns generated by specific sequencer and complex context modeling techniques to solve the problem of low compression ratio. However, the main drawbacks of these compressors are the problem of weak robustness which means unstable or even unavailable results of sequencing files and the problem of slow compression speed. Meanwhile, some compressors attempt to construct a fine-grained index structure to solve the problem of slow random access decompression speed. However, they solve the problem at the sacrifice of compression speed and at the expense of large index files, which makes them inefficient and impractical. Therefore, an efficient lossless compressor of quality scores with strong robustness, high compression ratio, fast compression and random access decompression speed is urgently needed and of great significance. Results: In this paper, based on the idea of maximizing the use of hardware resources, LCQS, a lossless compression tool specialized for quality scores, was proposed. It consists of four sequential processing steps: partitioning, indexing, packing and parallelizing. Experimental results reveal that LCQS outperforms all the other state-of-the-art compressors on all criteria except for the compression speed on the dataset SRR1284073. Furthermore, LCQS presents strong robustness on all the test datasets, with its acceleration ratios of compression speed increasing by up to 29.1x, its file size reducing by up to 28.78%, and its random access decompression speed increasing by up to 2.1x. Additionally, LCQS also exhibits strong scalability. That is, the compression speed increases almost linearly as the size of input dataset increases. Conclusion: The ability to handle all different kinds of quality scores and superiority in compression ratio and compression speed make LCQS a high-efficient and advanced lossless quality score compressor, along with its strength of fast random access decompression. Our tool LCQS can be downloaded from https://github.com/SCUT-CCNL/LCQS and freely available for non-commercial usage.

show abstract

Section: Parallelization Methods For Libzpaq Using Simd Techniquementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

LCQS: an efficient lossless compression tool of quality scores with random access functionality

Dong

2020

BMC Bioinformatics

View full text Add to dashboard Cite

show abstract

“…LFQC [33] Lossless non-reference based FASTQ compression algorithm by generating identifier fields systematically.…”

Section: Solution Contentsmentioning

confidence: 99%

BAQALC: Blockchain Applied Lossless Efficient Transmission of DNA Sequencing Data for Next Generation Medical Informatics

Lee¹,

Cho

Ikeno

et al. 2018

Applied Sciences

View full text Add to dashboard Cite

Due to the development of high-throughput DNA sequencing technology, genome-sequencing costs have been significantly reduced, which has led to a number of revolutionary advances in the genetics industry. However, the problem is that compared to the decrease in time and cost needed for DNA sequencing, the management of such large volumes of data is still an issue. Therefore, this research proposes Blockchain Applied FASTQ and FASTA Lossless Compression (BAQALC), a lossless compression algorithm that allows for the efficient transmission and storage of the immense amounts of DNA sequence data that are being generated by Next Generation Sequencing (NGS). Also, security and reliability issues exist in public sequence databases. For methods, compression ratio comparisons were determined for genetic biomarkers corresponding to the five diseases with the highest mortality rates according to the World Health Organization. The results showed an average compression ratio of approximately 12 for all the genetic datasets used. BAQALC performed especially well for lung cancer genetic markers, with a compression ratio of 17.02. BAQALC performed not only comparatively higher than widely used compression algorithms, but also higher than algorithms described in previously published research. The proposed solution is envisioned to contribute to providing an efficient and secure transmission and storage platform for next-generation medical informatics based on smart devices for both researchers and healthcare users.

show abstract

“…scRNA-seq utilizes the most common short-read mapping, as well as data storage and query procedures of common NGS applications. The Hadoop-based bioinformatics applications [61] , [62] , [63] , [64] , [65] , [66] , [67] , [68] , [69] , [70] , [71] , [72] , [73] , [74] , [75] , [76] , [77] , [78] , [79] , [80] , [81] , [82] , [83] , [84] , [85] , [86] , [87] , [88] , [89] , [90] , [91] are reviewed in Table 2 . To our best knowledge, there is no Hadoop application specially designed for scRNA-seq so far.…”

Section: Big Data—the Norm Of Ngs Technologymentioning

confidence: 99%

Single-Cell Transcriptome Study as Big Data

Lin

2016

Genomics, Proteomics &Amp; Bioinformatics

View full text Add to dashboard Cite

The rapid growth of single-cell RNA-seq studies (scRNA-seq) demands efficient data storage, processing, and analysis. Big-data technology provides a framework that facilitates the comprehensive discovery of biological signals from inter-institutional scRNA-seq datasets. The strategies to solve the stochastic and heterogeneous single-cell transcriptome signal are discussed in this article. After extensively reviewing the available big-data applications of next-generation sequencing (NGS)-based studies, we propose a workflow that accounts for the unique characteristics of scRNA-seq data and primary objectives of single-cell studies.

show abstract

LFQC: a lossless compression algorithm for FASTQ files

Abstract: rajasek@engr.uconn.edu.

Cited by 52 publications

References 15 publications

LCQS: an efficient lossless compression tool of quality scores with random access functionality

LCQS: an efficient lossless compression tool of quality scores with random access functionality

BAQALC: Blockchain Applied Lossless Efficient Transmission of DNA Sequencing Data for Next Generation Medical Informatics

Single-Cell Transcriptome Study as Big Data

Contact Info

Product

Resources

About