Container-based bioinformatics with Pachyderm

Novella, Jon Ander; Khoonsari, Payam Emami; Herman, Stephanie; Whitenack, Daniel; Capuccini, Marco; Burman, Joachim; Kultima, Kim; Spjuth, Ola

doi:10.1101/299032

Cited by 4 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Graphical tools like Galaxy [6, 7, 8] and Yabi [9] provide easy to use environments especially well-suited for scientists without scripting experience. Text-based tools like Snakemake [10], Nextflow [11], BPipe [12], Cuneiform [13] and Pachyderm [14] on the other hand, are implemented as Domain Specific Languages (DSLs), that can often provide a higher level of flexibility, at the expense of the ease of use of a graphical user interface. They can thus be well suited for “power users” with experience in scripting or programming.…”

Section: Resultsmentioning

confidence: 99%

SciPipe - A workflow library for agile development of complex and dynamic bioinformatics pipelines

Lampa

Dahlö

Alvarsson

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Background: The complex nature of biological data has driven the development of specialized sof ware tools. Scienti c work ow management systems simplify the assembly of such tools into pipelines, assist with job automation and aid reproducibility of analyses. Many contemporary work ow tools are specialized and not designed for highly complex work ows, such as with nested loops, dynamic scheduling and parametrization, which is common in e.g. machine learning.Findings: SciPipe is a work ow programming library implemented in the programming language Go, for managing complex and dynamic pipelines in bioinformatics, cheminformatics and other elds. SciPipe helps in particular with work ow constructs common in machine learning, such as extensive branching, parameter sweeps and dynamic scheduling and parametrization of downstream tasks. SciPipe builds on Flow-based programming principles to support agile development of work ows based on a library of self-contained, re-usable components. It supports running subsets of work ows for improved iterative development, and provides a data-centric audit logging feature that saves a full audit trace for every output le of a work ow, which can be converted to other formats such as HTML, TeX and PDF on-demand. The utility of SciPipe is demonstrated with a machine learning pipeline, a genomics, and a transcriptomics pipeline.Conclusions: SciPipe provides a solution for agile development of complex and dynamic pipelines, especially in machine leaning, through a exible programming API suitable for scientists used to programming or scripting.

show abstract

Section: Resultsmentioning

confidence: 99%

SciPipe - A workflow library for agile development of complex and dynamic bioinformatics pipelines

Lampa

Dahlö

Alvarsson

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The representative of virtualization is container technology [3] . The structure of a container is simple and consists of the underlying hardware, operating system, container engine, etc.…”

Section: Introductionmentioning

confidence: 99%

High Concurrency Response Strategy based on Kubernetes Horizontal Pod Autoscaler

Huo¹,

Li²,

Li³

et al. 2023

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

The automatic scaling of resources in the cluster can reduce costs while satisfying resource usage. This is one of the salient features of cloud computing. The current service scenarios are diverse, and the default policy of Horizontal Pod Autoscaling (HPA) is difficult to meet the demand. In the face of high concurrent traffic, the default strategy has the problem of long expansion time and slow speed. Based on HPA, this paper proposes the High Concurrency Response Strategy. It can cope with high concurrency usage scenarios and ensure that services can still be accessed in the face of large traffic concurrency. Compared with the default HPA, the customized HPA strategy reduces the number of service requests by over 20 ms or 19%.

show abstract

PhenoMeNal: Processing and analysis of Metabolomics data in the Cloud

Peters

Bradbury

Bergmann

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Background: Metabolomics is the comprehensive study of a multitude of small molecules to gain insight into an organism's metabolism. The research field is dynamic and expanding with applications across biomedical, biotechnological and many other applied biological domains. Its computationally-intensive nature has driven requirements for open data formats, data repositories and data analysis tools. However, the rapid progress has resulted in a mosaic of independent -and sometimes incompatible -analysis methods that are difficult to connect into a useful and complete data analysis solution. Findings: The PhenoMeNal (Phenome and Metabolome aNalysis) e-infrastructure provides a complete, workflow-oriented, interoperable metabolomics data analysis solution for a modern infrastructure-as-a-service (IaaS) cloud platform. PhenoMeNal seamlessly integrates a wide array of existing open source tools which are tested and packaged as Docker containers through the project's continuous integration process and deployed based on a kubernetes orchestration framework. It also provides a number of standardized, automated and published analysis workflows in the user interfaces Galaxy, Jupyter, Luigi and Pachyderm. Conclusions: PhenoMeNal constitutes a keystone solution in cloud infrastructures available for metabolomics. It provides scientists with a ready-to-use, workflow-driven, reproducible and shareable data analysis platform harmonizing the software installation and configuration through user-friendly web interfaces. The deployed cloud environments can be dynamically scaled to enable large-scale analyses which are interfaced through standard data formats, versioned, and have been tested for reproducibility and interoperability. The flexible implementation of PhenoMeNal allows easy adaptation of the infrastructure to other application areas and 'omics research domains.

show abstract

Container-based bioinformatics with Pachyderm

Cited by 4 publications

References 36 publications

SciPipe - A workflow library for agile development of complex and dynamic bioinformatics pipelines

SciPipe - A workflow library for agile development of complex and dynamic bioinformatics pipelines

High Concurrency Response Strategy based on Kubernetes Horizontal Pod Autoscaler

PhenoMeNal: Processing and analysis of Metabolomics data in the Cloud

Contact Info

Product

Resources

About