Benchmarking Short Sequence Mapping Tools

Abstract

Background: The development of next-generation sequencing instruments has led to the generation of millions of short sequences in a single run. The process of aligning these reads to a reference genome is time consuming and demands the development of fast and accurate alignment tools. However, the current proposed tools make different compromises between the accuracy and the speed of mapping. Moreover, many important aspects are overlooked while comparing the performance of a newly developed tool to the state of the art. Therefore, there is a need for an objective evaluation method that covers all the aspects. In this work, we introduce a benchmarking suite to extensively analyze sequencing tools with respect to various aspects and provide an objective comparison.Results: We applied our benchmarking tests on 9 well known mapping tools, namely, Bowtie, Bowtie2, BWA, SOAP2, MAQ, RMAP, GSNAP, Novoalign, and mrsFAST (mrFAST) using synthetic data and real RNA-Seq data. MAQ and RMAP are based on building hash tables for the reads, whereas the remaining tools are based on indexing the reference genome. The benchmarking tests reveal the strengths and weaknesses of each tool. The results show that no single tool outperforms all others in all metrics. However, Bowtie maintained the best throughput for most of the tests while BWA performed better for longer read lengths. The benchmarking tests are not restricted to the mentioned tools and can be further applied to others. Conclusion:The mapping process is still a hard problem that is affected by many factors. In this work, we provided a benchmarking suite that reveals and evaluates the different factors affecting the mapping process. Still, there is no tool that outperforms all of the others in all the tests. Therefore, the end user should clearly specify his needs in order to choose the tool that provides the best results.

From next-generation resequencing reads to a high-quality variant data set

“…Tools have been specifically designed to manipulate SAM/BAM files (for example, to quickly sort, merge or retrieve alignments), including the widely used SAMtools (Li et al, 2009b) and Picard. Several benchmarking studies have empirically compared short read alignment methods with respect to various metrics (that is, runtime, sensitivity and accuracy) using both simulated and real data sets from different organisms (Holtgrewe et al, 2011;Ruffalo et al, 2011;Fonseca et al, 2012;Lindner and Friedel, 2012;Schbath et al, 2012;Hatem et al, 2013;Caboche et al, 2014;Shang et al, 2014;Highnam et al, 2015). These analyses demonstrated that results depend strongly on the properties of the input data, and thus there is no single method best suited for all scenarios.…”

merAligner: A Fully Parallel Sequence Aligner

“…A thorough survey of sequence aligners is beyond the scope of our work and we refer the reader to [18], [19], [20], [21], [22], [23]. We primarily focus on parallel sequence mapping tools and relevant methods in this section.…”

Toward efficient and reliable genome analysis using main-memory database systems

“…Hatem et al show in a recent performance benchmark that current state-of-the-art sequence alignment tools can align 1 billion base pairs in one hour [7]. In contrast, current DNA sequencers have already a throughput of 2.5 billion base pairs per hour [12].…”