On-Demand Indexing for Referential Compression of DNA Sequences

Abstract

The decreasing costs of genome sequencing is creating a demand for scalable storage and processing tools and techniques to deal with the large amounts of generated data. Referential compression is one of these techniques, in which the similarity between the DNA of organisms of the same or an evolutionary close species is exploited to reduce the storage demands of genome sequences up to 700 times. The general idea is to store in the compressed file only the differences between the to-be-compressed and a well-known reference sequence. In this paper, we propose a method for improving the performance of referential compression by removing the most costly phase of the process, the complete reference indexing. Our approach, called On-Demand Indexing (ODI) compresses human chromosomes five to ten times faster than other state-of-the-art tools (on average), while achieving similar compression ratios.

Vertical lossless genomic data compression tools for assembled genomes: A systematic literature review

“…There are several approaches to the implementation of Local Research behavior, the main ones being highlighted here (Fig 6 ): 1. Keep a pointer between reference and target (Fig 6A), and advance this pointer at the same time in reference and target for every match or mutation identified, as implemented by tool JDNA [84]. Another possibility, implemented by tool GDC 2.0, is to perform some punctual simple verifications before initiating the search for a larger segment, advancing the pointer in reference [34];…”

“…Local search, that looks for short and local matches and a greedy search, that looks for the longest possible matches. It is important to notice that some tools implement more than one Local Search strategy, as RLZ-Opt [74] and JDNA [84], in which it is dynamically determined, during mapping, which local search strategies will be applied. GeCo and GReEn tools, despite not executing a mapping phase, use local search strategy to create and keep their statistical models updated.…”

Pattern Matching for DNA Sequencing Data Using Multiple Bloom Filters

“…Further, there are statistical methods that achieve extremely good compression rates by generating probabilistic models based on genome datasets [5–7]. The fourth category is referential compression where any repeated sequence in an input dataset is replaced with a reference to one or more external DNA sequences [8–11]. A data structure plays a critical role in any algorithm designed for achieving good compression ratios, fast searching of patterns inside sequences, or both.…”

From Data Islands to Sharing Data in the Cloud: the Evolution of Data Integration in Biological Data Repositories

“…Future work encompasses adding mechanisms for improved privacy-protection and compression of biological data [Alves et al 2015] to strengthen the overall security and efficiency of the system.…”