On-demand virtual research environments using microservices

Capuccini, Marco; Larsson, Anders; Carone, Matteo; Novella, Jon Ander; Sadawi, Noureddin; Gao, Jianliang; Toor, Salman; Spjuth, Ola

doi:10.7717/peerj-cs.232

Cited by 16 publications

(10 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Docker also provides Kubernetes as part of their Enterprise solutions (and even now the community ones). To enable convenient instantiation of a complete virtual infrastructure, we developed KubeNow (https://github.com/kubenow/KubeNow) (Capuccini et al , 2018) which includes instantiation of compute nodes, shared file system storage, networks, configure DNS, operating system, container implementation and orchestration tools, including Kubernetes, on a local computer or server. In order to deploy applications, we used two main classes of services: long-lasting services, and compute jobs.…”

Section: Methodsmentioning

confidence: 99%

“…The PhenoMeNal consortium maintains a web portal (https://portal.phenomenal-h2020.eu) providing a GUI for launching VREs using KubeNow (Capuccini et al , 2018) on a selection of the largest public cloud providers, including Amazon Web Services, Microsoft Azure and Google Cloud Platform, or on private OpenStack-based installations. The Wiki containing documentation is also hosted on GitHub https://github.com/phnmnl/phenomenal-h2020/wiki.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Interoperable and scalable data analysis with microservices: applications in metabolomics

et al. 2019

Self Cite

View full text Add to dashboard Cite

Motivation Developing a robust and performant data analysis workflow that integrates all necessary components whilst still being able to scale over multiple compute nodes is a challenging task. We introduce a generic method based on the microservice architecture, where software tools are encapsulated as Docker containers that can be connected into scientific workflows and executed using the Kubernetes container orchestrator. Results We developed a Virtual Research Environment (VRE) which facilitates rapid integration of new tools and developing scalable and interoperable workflows for performing metabolomics data analysis. The environment can be launched on-demand on cloud resources and desktop computers. IT-expertise requirements on the user side are kept to a minimum, and workflows can be re-used effortlessly by any novice user. We validate our method in the field of metabolomics on two mass spectrometry, one nuclear magnetic resonance spectroscopy and one fluxomics study. We showed that the method scales dynamically with increasing availability of computational resources. We demonstrated that the method facilitates interoperability using integration of the major software suites resulting in a turn-key workflow encompassing all steps for mass-spectrometry-based metabolomics including preprocessing, statistics and identification. Microservices is a generic methodology that can serve any scientific discipline and opens up for new types of large-scale integrative science. Availability and implementation The PhenoMeNal consortium maintains a web portal (https://portal.phenomenal-h2020.eu) providing a GUI for launching the Virtual Research Environment. The GitHub repository https://github.com/phnmnl/ hosts the source code of all projects. Supplementary information Supplementary data are available at Bioinformatics online.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Interoperable and scalable data analysis with microservices: applications in metabolomics

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Emami Khoonsari and colleagues 25 describe an approach to delivering a data analysis system for metabolomics through the use of a microservice architecture deployed on-demand as a set of containers (Docker) using an orchestration framework (Kubernetes). The role of microservices alongside on-demand resource allocation for VREs is identified in Capuccini and colleagues, 26 where a development methodology is described. The underlying principles include the use of continuous integration/continuous deployment (CI/CD) as a VRE collaboratively evolves, using infrastructure-as-code mechanisms for infrastructure provision and automated deployment tools for VREs.…”

Section: Resultsmentioning

confidence: 99%

Tackling the Challenges of 21st-Century Open Science and Beyond: A Data Science Lab Approach

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Collaborations were established with EGI [43] and Indigo Datacloud [44] infrastructure providers and initiatives [45, 46] to ensure that PhenoMeNal uses technologies that are well supported and ensure their widespread usage, continuity, and further development. For example, the development of KubeNow and contributions to the Galaxy and Workflow4Metabolomics community are essential for PhenoMeNal [47]. Core development will continue on GitHub and is fostered by collaborations with tool developers.…”

Section: Resultsmentioning

confidence: 99%

PhenoMeNal: processing and analysis of metabolomics data in the cloud

et al. 2018

Self Cite

View full text Add to dashboard Cite

BackgroundMetabolomics is the comprehensive study of a multitude of small molecules to gain insight into an organism's metabolism. The research field is dynamic and expanding with applications across biomedical, biotechnological, and many other applied biological domains. Its computationally intensive nature has driven requirements for open data formats, data repositories, and data analysis tools. However, the rapid progress has resulted in a mosaic of independent, and sometimes incompatible, analysis methods that are difficult to connect into a useful and complete data analysis solution.FindingsPhenoMeNal (Phenome and Metabolome aNalysis) is an advanced and complete solution to set up Infrastructure-as-a-Service (IaaS) that brings workflow-oriented, interoperable metabolomics data analysis platforms into the cloud. PhenoMeNal seamlessly integrates a wide array of existing open-source tools that are tested and packaged as Docker containers through the project's continuous integration process and deployed based on a kubernetes orchestration framework. It also provides a number of standardized, automated, and published analysis workflows in the user interfaces Galaxy, Jupyter, Luigi, and Pachyderm.ConclusionsPhenoMeNal constitutes a keystone solution in cloud e-infrastructures available for metabolomics. PhenoMeNal is a unique and complete solution for setting up cloud e-infrastructures through easy-to-use web interfaces that can be scaled to any custom public and private cloud environment. By harmonizing and automating software installation and configuration and through ready-to-use scientific workflow user interfaces, PhenoMeNal has succeeded in providing scientists with workflow-driven, reproducible, and shareable metabolomics data analysis platforms that are interfaced through standard data formats, representative datasets, versioned, and have been tested for reproducibility and interoperability. The elastic implementation of PhenoMeNal further allows easy adaptation of the infrastructure to other application areas and ‘omics research domains.

show abstract

On-demand virtual research environments using microservices

Cited by 16 publications

References 43 publications

Interoperable and scalable data analysis with microservices: applications in metabolomics

Interoperable and scalable data analysis with microservices: applications in metabolomics

Tackling the Challenges of 21st-Century Open Science and Beyond: A Data Science Lab Approach

PhenoMeNal: processing and analysis of metabolomics data in the cloud

Contact Info

Product

Resources

About