Toil enables reproducible, open source, big biomedical data analyses

Vivian, John; Av, Rao; Nothaft, Frank Austin; Ketchum, Christopher; Armstrong, Joel; Novak, Adam M.; Pfeil, Jacob; Narkizian, Jake; Deran, Alden; Musselman-Brown, Audrey; Schmidt, Hannes; Amstutz, Peter; Craft, Brian; Goldman, Mary; Rosenbloom, Kate R.; Cline, Melissa; O’Connor, Brian D.; Hanna, Megan; Birger, Chet; Kent, W. James; Patterson, David A.; Joseph, Anthony D.; Zhu, Jingchun; Zaranek, Alexander Wait; Getz, Gad; Haussler, David; Paten, Benedict

doi:10.1038/nbt.3772

Cited by 1,007 publications

(877 citation statements)

References 14 publications

Supporting

Mentioning

811

Contrasting

Unclassified

Order By: Relevance

“…It is essential that experiments are analyzed in a reproducible manner. Computational workflow systems allow for automation of analysis pipelines, that scale from personal computers to HPC and cloud environments (Koster and Rahmann 2012;Di Tommaso et al 2017;Vivian et al 2017). While standardized solutions exist for installation of software and analysis tools ("Pip," n.d.; "Bioconda," n.d.), data download often has to either be performed manually or be scripted on a caseby-case basis.…”

Section: Discussionmentioning

confidence: 99%

genomepy: download genomes the easy way

Heeringen¹

2017

JOSS

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

genomepy: download genomes the easy way

Heeringen¹

2017

JOSS

View full text Add to dashboard Cite

“…Both the pipeline and associated documentation can be found at https://github.com/ComparativeGenomicsToolkit/ Comparative-Annotation-Toolkit. CAT is constructed using the Luigi (https:// github.com/spotify/luigi) workflow manager, with Toil [154] used for computationally intensive steps that work best when submitted to a compute cluster.…”

Section: Methodsmentioning

confidence: 99%

Comparative Annotation Toolkit (CAT) - simultaneous clade and personal genome annotation

Fiddes

Armstrong

Diekhans

et al. 2017

Preprint

Self Cite

View full text Add to dashboard Cite

The recent introductions of low-cost, long-read, and read-cloud sequencing technologies coupled with intense efforts to develop efficient algorithms have made affordable, high-quality de novo sequence assembly a realistic proposition. The result is an explosion of new, ultra-contiguous genome assemblies. To compare these genomes we need robust methods for genome annotation. We describe the fully open source Comparative Annotation Toolkit (CAT), which provides a flexible way to simultaneously annotate entire clades and identify orthology relationships. We show that CAT can be used to improve annotations on the rat genome, annotate the great apes, annotate a diverse set of mammals, and annotate personal, diploid human genomes. We demonstrate the resulting discovery of novel genes, isoforms and structural variants, even in genomes as well studied as rat and the great apes, and how these annotations improve cross-species RNA expression experiments.

show abstract

“…Both the Dockerfile and the resulting container can be moved between machines without installing additional software, and the container can be rerun later with the exact same libraries. Containers can be orchestrated for parallel execution using, for example, Kubernetes (https://kubernetes.io/) or Docker Swarm (https://github.com/docker/swarm), and there are now multiple pipelining tools that use Docker or provide Docker container support including Nextflow [26], Toil [22], Pachyderm (http://www.pachyderm.io/), Luigi (https:// github.com/spotify/luigi) [27], Rabix/bunny [28], and our own walrus system (http://github.com/fjukstad/walrus).…”

Section: Current Trendsmentioning

confidence: 99%

“…Toil is a workflow software to run scientific workflows on a large scale in cloud or high-performance computing (HPC) environments [22]. It is designed for large-scale analysis pipelines such as The Cancer Genome Atlas (TCGA) [23] best practices pipeline for calculating gene-and isoformlevel expression values from RNA-seq data.…”

Section: Toil: Tcga Rna-seq Reference Pipelinementioning

confidence: 99%

See 1 more Smart Citation

A Review of Scalable Bioinformatics Pipelines

Fjukstad

Bongo

2017

Data Sci. Eng.

View full text Add to dashboard Cite

Scalability is increasingly important for bioinformatics analysis services, since these must handle larger datasets, more jobs, and more users. The pipelines used to implement analyses must therefore scale with respect to the resources on a single compute node, the number of nodes on a cluster, and also to cost-performance. Here, we survey several scalable bioinformatics pipelines and compare their design and their use of underlying frameworks and infrastructures. We also discuss current trends for bioinformatics pipeline development.

show abstract

Toil enables reproducible, open source, big biomedical data analyses

Cited by 1,007 publications

References 14 publications

genomepy: download genomes the easy way

genomepy: download genomes the easy way

Comparative Annotation Toolkit (CAT) - simultaneous clade and personal genome annotation

A Review of Scalable Bioinformatics Pipelines

Contact Info

Product

Resources

About