Artificial and natural duplicates in pyrosequencing reads of metagenomic data

Abstract

BackgroundArtificial duplicates from pyrosequencing reads may lead to incorrect interpretation of the abundance of species and genes in metagenomic studies. Duplicated reads were filtered out in many metagenomic projects. However, since the duplicated reads observed in a pyrosequencing run also include natural (non-artificial) duplicates, simply removing all duplicates may also cause underestimation of abundance associated with natural duplicates.ResultsWe implemented a method for identification of exact and nearly identical duplicates from pyrosequencing reads. This method performs an all-against-all sequence comparison and clusters the duplicates into groups using an algorithm modified from our previous sequence clustering method cd-hit. This method can process a typical dataset in ~10 minutes; it also provides a consensus sequence for each group of duplicates. We applied this method to the underlying raw reads of 39 genomic projects and 10 metagenomic projects that utilized pyrosequencing technique. We compared the occurrences of the duplicates identified by our method and the natural duplicates made by independent simulations. We observed that the duplicates, including both artificial and natural duplicates, make up 4-44% of reads. The number of natural duplicates highly correlates with the samples' read density (number of reads divided by genome size). For high-complexity metagenomic samples lacking dominant species, natural duplicates only make up <1% of all duplicates. But for some other samples like transcriptomic samples, majority of the observed duplicates might be natural duplicates.ConclusionsOur method is available from http://cd-hit.org as a downloadable program and a web server. It is important not only to identify the duplicates from metagenomic datasets but also to distinguish whether they are artificial or natural duplicates. We provide a tool to estimate the number of natural duplicates according to user-defined sample types, so users can decide whether to retain or remove duplicates in their projects.

Deep Sequencing Reveals Mutagenic Effects of Ribavirin during Monotherapy of Hepatitis C Virus Genotype 1-Infected Patients

“…Nevertheless, it is critical to differentiate minor variants from technical errors associated with library preparation and the sequencing process. By sequencing a clonal fragment, we determined for the 454 platform and the Illumina platform error rates in good accordance with data from previous studies (39,41,43,45,55). Although specialized software for additional error correction exists (33,40), we refrained from employing it in this setting due to the risk of discarding genuine minority variants.…”

“…The insert of one sequenced plasmid clone was excised from the vector with the restriction enzyme EcoRI and was deep sequenced in parallel with the other PCR amplicon samples. In accordance with other studies (33,(39)(40)(41)(42)(43), the technical error rate was calculated by counting all nucleotide variants of the plasmid reads in the alignment that did not correspond to the sequence of the clone determined by Sanger sequencing. While insertion errors were subject to automatic removal during the mapping of the sequencing reads to the HCV-J reference genome (34), deletions with respect to the reference sequence were detected during the mapping and quantified as errors but excluded from all further analyses.…”

Genome sequencing of a single cell of the widely distributed marine subsurface Dehalococcoidia, phylum Chloroflexi

“…After mis-assigned reads were removed, the remaining Illumina reads were assembled using SPAdes assembler version 2.3.0 (Bankevich et al, 2012) (parameters: -k 21,33,45 -sc). The 454-pyrosequence reads were dereplicated using cd-hit-454 (Niu et al, 2010) with a 98% similarity cutoff and assembled using GS De Novo Assembler version 2.6 (gsAssembler, Roche) (parameters: -mi 98 -ml 50). The two assemblies were finally combined using Sequencher version 5.0.1 (Genecodes) (Lloyd et al, 2013).…”

Next-generation sequencing reveals phylogeographic structure and a species tree for recent bird divergences

“…Error rates averaged 0.1%, lower than some others NGS studies using 454 sequencing (Niu et al, 2010), which might be attributed to the fact that called alleles went through a multi-level process of screening and error filtering in the analytical pipeline. Of the few errors, most involved calling individuals homozygous when they were actually heterozygotes.…”