Reference-free compression of high throughput sequencing data with a probabilistic de Bruijn graph

Abstract

BackgroundData volumes generated by next-generation sequencing (NGS) technologies is now a major concern for both data storage and transmission. This triggered the need for more efficient methods than general purpose compression tools, such as the widely used gzip method.ResultsWe present a novel reference-free method meant to compress data issued from high throughput sequencing technologies. Our approach, implemented in the software Leon, employs techniques derived from existing assembly principles. The method is based on a reference probabilistic de Bruijn Graph, built de novo from the set of reads and stored in a Bloom filter. Each read is encoded as a path in this graph, by memorizing an anchoring kmer and a list of bifurcations. The same probabilistic de Bruijn Graph is used to perform a lossy transformation of the quality scores, which allows to obtain higher compression rates without losing pertinent information for downstream analyses.Conclusions Leon was run on various real sequencing datasets (whole genome, exome, RNA-seq or metagenomics). In all cases, LEON showed higher overall compression ratios than state-of-the-art compression software. On a C. elegans whole genome sequencing dataset, LEON divided the original file size by more than 20. Leon is an open source software, distributed under GNU affero GPL License, available for download at http://gatb.inria.fr/software/leon/.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-015-0709-7) contains supplementary material, which is available to authorized users.

Nucleotide Archival Format (NAF) enables efficient lossless reference-free compression of DNA sequences

“…In 1993 the first specialized DNA compressor was proposed (Grumbach and Tahi, 1993). Since then, numerous DNA compressors were developed (e.g., Cao et al, 2007, Li et al, 2013, Benoit et al, 2015, Al-Okaily et al, 2017. In our experience only two compressors pass the practicality threshold: DELIMINATE (Mohammed et al, 2012) and MFCompress (Pinho and Pratas, 2014).…”

Bifrost – Highly parallel construction and indexing of colored and compacted de Bruijn graphs

“…Although SMS (Single Molecule Sequencing) technologies (Rang et al, 2018;Rhoads and Au, 2015) have re-introduced the OLC framework as the method of choice to assemble long and erroneous reads (Koren et al, 2017;Li, 2016;Chin et al, 2016;Kamath et al, 2017), de Bruijn graph based methods are nonetheless used to assemble and correct long reads (Salmela and Rivals, 2014;Ruan and Li, 2019). Overall, the de Bruijn graphs have found widespread use for a variety of problems such as de novo transcriptome assembly (Robertson et al, 2010), variant calling (Uricaru et al, 2015), short read compression (Benoit et al, 2015), short read correction (Limasset et al, 2019), long read correction (Salmela and Rivals, 2014) and short read mapping (Liu et al, 2016) to name a few. The colored de Bruijn graph is a variant of the de Bruijn graph which keeps track of the source of each vertex in the graph (Iqbal et al, 2012).…”

Improving Bloom Filter Performance on Sequence Data Using $$k$$ -mer Bloom Filters

“…For example, sequence assembly algorithms use k-mers as nodes in the de Bruijn graph (Zerbino and Birney, 2008;Pell et al, 2012), metagenomic sample diversity can be quantified by comparing the sample's k-mer content against a database (Wood and Salzberg, 2014), k-mer content derived from RNA-seq reads can inform gene expression estimation procedures (Patro et al, 2014), and k-mer-based algorithms can dramatically improve compression of sequence (Rozov et al, 2014;Benoit et al, 2015) and quality values (Yu et al, 2014).…”