EDAR: An Efficient Error Detection and Removal Algorithm for Next Generation Sequencing Data

Abstract

Genomic sequencing techniques introduce experimental errors into reads which can mislead sequence assembly efforts and complicate the diagnostic process. Here we present a method for detecting and removing sequencing errors from reads generated in genomic shotgun sequencing projects prior to sequence assembly. For each input read, the set of all length k substrings (k-mers) it contains are calculated. The read is evaluated based on the frequency with which each k-mer occurs in the complete data set (k-count). For each read, k-mers are clustered using the variable-bandwidth mean-shift algorithm. Based on the k-count of the cluster center, clusters are classified as error regions or non-error regions. For the 23 real and simulated data sets tested (454 and Solexa), our algorithm detected error regions that cover 99% of all errors. A heuristic algorithm is then applied to detect the location of errors in each putative error region. A read is corrected by removing the errors, thereby creating two or more smaller, error-free read fragments. After performing error removal, the error-rate for all data sets tested decreased (∼35-fold reduction, on average). EDAR has comparable accuracy to methods that correct rather than remove errors and when the error rate is greater than 3% for simulated data sets, it performs better. The performance of the Velvet assembler is generally better with error-removed data. However, for short reads, splitting at the location of errors can be problematic. Following error detection with error correction, rather than removal, may improve the assembly results.

Analysis of the Evolution and Structure of a Complex Intrahost Viral Population in Chronic Hepatitis C Virus Mapped by Ultradeep Pyrosequencing

“…In stage 1, the set of k-mers (substring of fixed length k) of reads from the processed data set is calculated and the distribution of frequencies of k-mers is analyzed (31). It was previously observed that the frequencies of erroneous and correct k-mers follow different distributions (32)(33)(34). Based on this fact, the error threshold is calculated as the minimal frequency of k-mers separating two different distributions.…”

Deep Sequencing Data Analysis: Challenges and Solutions

“…Available quality control software allow the user to completely remove these duplicates (FASTX -toolkit; or mark them for downstream analysis consideration (PICARD). Recently various algorithms utilizing suffix tree data structures were developed for sequencing error correction (Kelley et al, 2010;Zhao et al, 2010). A common procedure in the pre-analysis process, following initial quality control, and prior to sequence duplication removal, is the compulsory tag / adapter removal (Lassmann et al, 2009;Schmieder et al, 2010) and optional quality trimming.…”

Improved Error Correction of NGS Data

“…[99,11] On the other hand, the exact error rate for real data can only be estimated. [53] Gain/Specificity/Sensitivity…”

“…EDAR [99] removes low quality reads and, from the remaining data, calculates the coverage for all possible k-mers. Using the variable bandwidth mean-shift method [100] for each read, EDAR clusters the k-mers and set each cluster as erroneous or correct using a threshold derived from the normalized distribution of the coverage.…”

“…When dealing with very long reads like Pacific Biosciences with a high percentage of errors [71], along with considering trimming their ends, an additional approach is to split the long reads into smaller, high quality segments. The authors of [99] do not correct the spurious bases, opting instead to split the read at the locus of the error and to remove the faulty base. They argue that for a further assembly step based on k-mers such as Velvet [196], their method should not pose any problems.…”