HUBzero and Pegasus: integrating scientific workflows into science gateways

McLennan, Michael; Clark, Steven M.; Deelman, Ewa; Rynge, Mats; Vahi, Karan; McKenna, Frank; Kearney, Derrick; Song, Carol X.

doi:10.1002/cpe.3257

Cited by 25 publications

(15 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Similarly, the computing management and monitoring functions are provided by a single component (OpenDreams) in the SINAPAD SG [11]. And in [21] coordination and provenance functions are provided by the Pegasus WfMS. Putting all these SG implementation variations into perspective, in addition to their function, would be also very useful for newcomers SG development.…”

Section: Discussionmentioning

confidence: 99%

“…Pegasus is a WfMS that maps abstract workflows descriptions onto cyberinfrastructures [6]. As described in [21], the integration of these two platforms provides a Web-based venue for building, sharing, and delivering tools for education and research, and for executing them on a number of computing infrastructures. Data management functions are provided by HTCondor I/O using protocols and systems such as GridFTP, S3, and SRM.…”

Section: Hubzero + Pegasusmentioning

confidence: 99%

See 1 more Smart Citation

Science Gateway Canvas

Shahand

Kampen

Olabarriaga

2015

Proceedings of the 1st Workshop on the Science of Cyberinfrastructure: Research, Experience, Applications and Models

View full text Add to dashboard Cite

Science Gateways (SGs) have emerged as systems that facilitate access to cyberinfrastructures. There is a growing interest in the exploitation and development of SGs. However, it remains challenging to understand and design SG with the required properties because of the complex nature of SGs. Additionally, it is difficult to decide upon frameworks that can be used to build them. In this paper we propose the Science Gateway Canvas, a business reference model for SGs that embodies the common SG functions and their organization into groups and categories. We used the Science Gateway Canvas for a systematic analysis and comparison of the functions offered by a selection of available frameworks that are used to build SGs. This illustrated the applicability of the Science Gateway Canvas as a comprehensive and generic reference model for understanding SGs. The canvas can also be used for systematic analysis by scientists who are searching for an existing SG that supports their research goal or developers who want to determine the functional requirements for a new SG.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Hubzero + Pegasusmentioning

confidence: 99%

Science Gateway Canvas

Shahand

Kampen

Olabarriaga

2015

Proceedings of the 1st Workshop on the Science of Cyberinfrastructure: Research, Experience, Applications and Models

View full text Add to dashboard Cite

show abstract

“…The result is that nanoHUB now contributes to and benefits from development efforts to expand and improve the functionality of the core infrastructure, known as HUBzero. The nanoHUB cyberinfrastructure does not operate in a vacuum, but rather takes the opportunity to leverage and incorporate appropriate technologies, such as Pegasus workflow management tools [5,6], to the benefit of the nanoHUB users and tool developers. The nanoHUB and HUBzero teams approach incorporation of other existing technologies as driven by use cases and leveragability.…”

Section: The Nanohub Communitymentioning

confidence: 99%

nanoHUB.org: Experiences and Challenges in Software Sustainability for a Large Scientific Community

Zentner¹,

Zentner²,

Farnsworth³

et al. 2014

Journal of Open Research Software

Self Cite

View full text Add to dashboard Cite

BackgroundThe Network for Computational Nanotechnology Cyberinfrastructure Node (NCN CI) is responsible for the operation and support of the cyber-platform nanoHUB.org, serving a large scientific community centered around the nanoscience and nanotechnology fields. The word "community" is constantly on our minds. One definition of community is a "body of persons of common and especially professional interests scattered through a larger society. 1 " For us, this means we have a challenge of supporting a variety of community members, with varying viewpoints and needs; generally many wish to participate and utilize nanoHUB while a significantly smaller subset contribute to our online facility for that community. The staunchness of policy we set, the degree to which we enforce it, and the features we choose to make available are under constant examination: Are we making decisions that selectively encourage some members of the community while alienating others? Is the community we serve only those who agree with the policies we set, or do we need to look at setting policies that do not unnecessarily exclude well-respected members of the nanoscience and nanotechnology fields? nanoHUB has a significant ten year history of data and experience. nanoHUB users exceed 260,000 annually and access a portfolio of over 4000 resources contributed by over 1,000 authors, including over 300 simulation tools contributed by 411 software authors and developers. Such broad use and a large, vibrant community provide continued opportunities for growth, but careful management of policies and processes is necessary to anticipate and meet associated challenges.

show abstract

“…Several systems report the existence of repositories for workflows or experiments, like the myExperiment [23] repository, which currently supports inputs from several systems (Taverna, Galaxy and Kepler), or HUBzero [37] a software platform to support collaborations that is able to launch Pegasus workflows.…”

Section: Workflow Management Systems: An Overviewmentioning

confidence: 99%

Workflows for science: a challenge when facing the convergence of HPC and Big Data

Badía

Ayguadé

Labarta

2017

JSFI

View full text Add to dashboard Cite

Workflows have been traditionally a mean to describe and implement the computing experiments, usually parametric studies and explorations searching for the best solution, that scientific researchers want to perform. A workflow is not only the computing application, but a way of documenting a process. Science workflows may be of very different nature depending on the area of research, matching the actual experiment that the scientist want to perform. Workflow Management Systems are environments that offer the researchers tools to define, publish, execute and document their workflows.In some cases, the science workflows are used to generate data; in other cases are used to analyse existing data; only in a few cases, workflows are used both to generate and analyse data. The design of experiments is in some cases generated blindly, without a clear idea of which points are relevant to be computed/simulated, ending up with huge amount of computation that is performed following a brute-force strategy.However, the evolution of systems and the large amount of data generated by the applications require an in-situ analysis of the data, thus requiring new solutions to develop workflows that includes both the simulation/computational part and the analytic part. What is more, the fact that both components, computation and analytics, can be run together will enable the possibility of defining more dynamic workflows, with new computations being decided by the analytics in a more efficient way.The first part of the paper will review current approaches that a set of scientific communities follow in the development of their workflows. This paper does not intent to be exhaustive in the compilation of different approaches available to develop and deploy workflows. We focus on the Workflow Management Systems used by a set of scientific communities and their representative use cases, with the objective of understanding their different needs and requirements. The second part of the paper proposes a new software architecture to develop a new family of end-to-end workflows that enables the management of dynamic workflows composed of simulations, analytics and visualization, including inputs/outputs from streams.Keywords: workflows, scientific applications, Big Data. IntroductionWorkflows appeared last century and have been used in the manufacturing industry as a mean to optimize their processes. Examples of traditional (non-IT) workflows can be found in the assembly lines, i.e., the Ford Model T assembly line standardized the production processes and was the first continuous delivery pipeline for the automotive industry. This process reduced the costs of manufacturing from $850 to $260 in 1924.The time and motion studies defined by Taylor [52] and Gilbreth [41] had significant impact in the manufacturing processes. These studies proposed to break manufacturing activities into small, simple steps, to determine with accuracy the amount of time required to perform each of the steps. Then, the sequence of movements taken by the employee has...

show abstract

HUBzero and Pegasus: integrating scientific workflows into science gateways

Cited by 25 publications

References 29 publications

Science Gateway Canvas

Science Gateway Canvas

nanoHUB.org: Experiences and Challenges in Software Sustainability for a Large Scientific Community

Workflows for science: a challenge when facing the convergence of HPC and Big Data

Contact Info

Product

Resources

About