A big data approach to metagenomics for all-food-sequencing

Kobus, Robin; Abuín, José Manuel; Müller, André C.; Hellmann, Sören Lukas; Pichel, Juan C.; Pena, Tomás F.; Hildebrandt, Andreas; Hankeln, Thomas; Schmidt, Bertil

doi:10.1186/s12859-020-3429-6

Cited by 24 publications

(31 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…consequently lowering the classification speed. Alternatively, it is possible to use the Apache Spark based version of this approach, named MetaCacheSpark, as presented in [25].…”

Section: Plos Onementioning

confidence: 99%

“…The following parameters have been used during the experiments, as they are the same as used in [25]. For building:…”

Section: Parameters Used During Executionsmentioning

confidence: 99%

“…As for the memory consumption, there is relevant reduction of about half the usage in the MPI version, when using only 8 processes. For example, for the biggest database AFS31RS90, according to [25], the memory consumed for MetaCache during this phase is 117 GB, while the MPI version is able to decrease it to a peak memory of 41.50 GB using 8 cores and 16 threads per process. This indicates that, with these input data, MetaCache-MPI can run in typical computing clusters with a typical amount of memory per node (64-128 GB) and, at the same time, get classification speeds higher than the ones obtained with the original version of MetaCache.…”

Section: Plos Onementioning

confidence: 99%

“…This accuracy has been tested throughout this work, where the authors have run a set of different experiments that have also been performed in [25]. In this way, results involving three different approaches were tested: one from MetaCache, the original sequential tool; other by using the Big Data engine Apache Spark; and the approach introduced in this work using MPI.…”

Section: Plos Onementioning

confidence: 99%

“…Recent publications [24,25], show that a k-mer-based exact matching approach can achieve high classification accuracy while being orders-of-magnitude faster than the alignment-based AFS pipeline. It relies on building a database of substrings of length k of each considered reference genome.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Big Data in metagenomics: Apache Spark vs MPI

et al. 2020

Self Cite

View full text Add to dashboard Cite

The progress of next-generation sequencing has lead to the availability of massive data sets used by a wide range of applications in biology and medicine. This has sparked significant interest in using modern Big Data technologies to process this large amount of information in distributed memory clusters of commodity hardware. Several approaches based on solutions such as Apache Hadoop or Apache Spark, have been proposed. These solutions allow developers to focus on the problem while the need to deal with low level details, such as data distribution schemes or communication patterns among processing nodes, can be ignored. However, performance and scalability are also of high importance when dealing with increasing problems sizes, making in this way the usage of High Performance Computing (HPC) technologies such as the message passing interface (MPI) a promising alternative. Recently, MetaCacheSpark, an Apache Spark based software for detection and quantification of species composition in food samples has been proposed. This tool can be used to analyze high throughput sequencing data sets of metagenomic DNA and allows for dealing with large-scale collections of complex eukaryotic and bacterial reference genome. In this work, we propose MetaCache-MPI, a fast and memory efficient solution for computing clusters which is based on MPI instead of Apache Spark. In order to evaluate its performance a comparison is performed between the original single CPU version of MetaCache, the Spark version and the MPI version we are introducing. Results show that for 32 processes, MetaCache-MPI is 1.65× faster while consuming 48.12% of the RAM memory used by Spark for building a metagenomics database. For querying this database, also with 32 processes, the MPI version is 3.11× faster, while using 55.56% of the memory used by Spark. We conclude that the new MetaCache-MPI version is faster in both building and querying the database and uses less RAM memory, when compared with MetaCacheSpark, while keeping the accuracy of the original implementation.

show abstract

“…consequently lowering the classification speed. Alternatively, it is possible to use the Apache Spark based version of this approach, named MetaCacheSpark, as presented in [25].…”

Section: Plos Onementioning

confidence: 99%

“…The following parameters have been used during the experiments, as they are the same as used in [25]. For building:…”

Section: Parameters Used During Executionsmentioning

confidence: 99%

Section: Plos Onementioning

confidence: 99%

Section: Plos Onementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Big Data in metagenomics: Apache Spark vs MPI

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

Fish Traceability and Authenticity

GONZÁLEZ SOTELO,

GUNNLAUGSSON,

SCHRÖDER

et al. 2023

Current Challenges for the Aquatic Products Processing Industry

View full text Add to dashboard Cite

Optimizing a metabarcoding marker portfolio for species detection from complex mixtures of globally diverse fishes

Baetscher,

Locatelli,

Won

et al. 2023

Environmental DNA

View full text Add to dashboard Cite

DNA metabarcoding is used to enumerate and identify taxa in both environmental samples and tissue mixtures, but the effectiveness of particular markers depends on their sensitivity to the taxa involved. Using multiple primer sets that amplify different genes can mitigate biases in amplification efficiency, sequence resolution, and reference data availability, but few empirical studies have evaluated markers for complementary performance. Here, we assess the individual and joint performance of 22 markers for detecting species in a DNA pool of 98 species of marine and freshwater bony fishes from geographically and phylogenetically diverse origins. We find that a portfolio of four markers targeting 12S, 16S, and two regions of COI identifies 100% of reference taxa to family and nearly 60% to species. We then use these four markers to evaluate metabarcoding of heterogeneous tissue mixtures, using experimental fishmeal to test: (1) the tissue input threshold to ensure detection; (2) how read depth scales with tissue abundance; and (3) the effect of non‐target material in the mixture on recovery of target taxa. We consistently detect taxa that makeup >1% of fishmeal mixtures and can detect taxa at the lowest input level of 0.01%, but rare taxa (<1%) were detected inconsistently across markers and replicates. Read counts showed only a weak correlation with tissue input, suggesting they are not a reliable quantitative proxy for relative abundance. Despite the limitations arising from primer specificity and reference data availability, our results demonstrate that a modest portfolio of markers can perform well in detecting and identifying aquatic species in complex mixtures despite heterogeneity in tissue representation, phylogenetic affinities, and from a broad geographic range.

show abstract

A big data approach to metagenomics for all-food-sequencing

Cited by 24 publications

References 34 publications

Big Data in metagenomics: Apache Spark vs MPI

Big Data in metagenomics: Apache Spark vs MPI

Fish Traceability and Authenticity

Optimizing a metabarcoding marker portfolio for species detection from complex mixtures of globally diverse fishes

Contact Info

Product

Resources

About