Exploring many task computing in scientific workflows

Ogasawara, Eduardo; Oliveira, Daniel de; Chirigati, Fernando; Barbosa, Carlos Eduardo; Elias, Renato N.; Braganholo, Vanessa; Coutinho, Álvaro L. G. A.; Mattoso, Marta

doi:10.1145/1646468.1646470

Cited by 28 publications

(33 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hydra [17] is a middleware that provides a set of components to be included on the workflow specification of any SWfMS to control parallelism of activities following the MTC paradigm. Hydra is based on a homogeneous cluster environment and relies on a centralized scheduler (such as Torque [18]).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Data parallelism in bioinformatics workflows using Hydra

Coutinho

Ogasawara

Oliveira

et al. 2010

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Self Cite

View full text Add to dashboard Cite

Large scale bioinformatics experiments are usually composed by a set of data flows generated by a chain of activities (programs or services) that may be modeled as scientific workflows. Current Scientific Workflow Management Systems (SWfMS) are used to orchestrate these workflows to control and monitor the whole execution. It is very common in bioinformatics experiments to process very large datasets. In this way, data parallelism is a common approach used to increase performance and reduce overall execution time. However, most of current SWfMS still lack on supporting parallel executions in high performance computing (HPC) environments. Additionally keeping track of provenance data in distributed environments is still an open, yet important problem. Recently, Hydra middleware was proposed to bridge the gap between the SWfMS and the HPC environment, by providing a transparent way for scientists to parallelize workflow executions while capturing distributed provenance. This paper analyzes data parallelism scenarios in bioinformatics domain and presents an extension to Hydra middleware through a specific cartridge that promotes data parallelism in bioinformatics workflows. Experimental results using workflows with BLAST show performance gains with the additional benefits of distributed provenance support.

show abstract

Section: Introductionmentioning

confidence: 99%

“…Using Hydra, the MTC parallelism strategy can be registered, reused, and provenance may be uniformly gathered during the execution of workflows. In previous work [17,19], parameter sweep mechanisms were developed and explored using Hydra. However, Hydra still lacks on data parallelism mechanisms coupled to provenance facilities.…”

Section: Introductionmentioning

confidence: 99%

Data parallelism in bioinformatics workflows using Hydra

Coutinho

Ogasawara

Oliveira

et al. 2010

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…After this filter, the number of workflows was reduced to 36 workflows (C 36 2 = 630 comparisons). The machine in which SimiFlow was executed using Hydra middleware (Ogasawara et al 2009a) is the SGI Altix ICE 8200 with 64 nodes (Intel Xeon 8-core processor). The experiment used four nodes (32 cores).…”

Section: Resultsmentioning

confidence: 99%

Similarity-based workflow clustering

Silva¹,

Chirigati

Maia³

et al. 2011

JCIS

View full text Add to dashboard Cite

Scientists have been using scientific workflow management systems (SWfMS) to support scientific experiments. However, SWfMS expect a modeled workflow to be represented on its workflow language to be executed. The scientist does not have an assistance or guidance to obtain a modeled workflow. Experiment lines, which are a novel approach to deal with these limitations, allow for the abstract representation and systematic composition of experiments. Since there are many scientific workflows already modeled and successfully executed, they can be used to leverage the construction of new abstract representations. These previous experiments can be helpful by identifying scientific workflow clusters that are generated according to similarity criteria. This paper proposes SimiFlow, which is an architecture for similarity-based comparison and clustering to build experiment lines following a bottom-up approach.

show abstract

“…Even though the workflow execution log could be browsed, this is far from provenance data query. Our previous works in supporting workflow scientists from bioinformatics [12] and numerical methods [13] have led us to develop services to query provenance data during the execution [14]. We define this type of provenance as runtime provenance data.…”

Section: Introductionmentioning

confidence: 99%

Using Domain-Specific Data to Enhance Scientific Workflow Steering Queries

Gonçalves

Oliveira

Ocaña

et al. 2012

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. In scientific workflows, provenance data helps scientists in understanding, evaluating and reproducing their results. Provenance data generated at runtime can also support workflow steering mechanisms. Steering facilities for workflows is considered a challenge due to its dynamic demands during execution. To steer, for example, scientists should be able to suspend (or stop) a workflow execution when the approximate solution meets (or deviates) preset criteria. These criteria are commonly evaluated based on provenance data (execution data) and domain-specific data. We claim that the final decision on whether to interfere on the workflow execution may only become feasible when workflows can be steered by scientists using provenance data enriched with domain-specific data. In this paper we propose an approach based on specialized software components, named Data Extractor (DE), to acquire domain-specific data from data files produced during a scientific workflow execution. DE gathers domain-specific data from produced data files and associates it to existing provenance data on the provenance repository. We have evaluated the proposed approach using a real bioinformatics workflow for comparative genomics executed in SciCumulus cloud workflow parallel engine.

show abstract

Exploring many task computing in scientific workflows

Cited by 28 publications

References 31 publications

Data parallelism in bioinformatics workflows using Hydra

Data parallelism in bioinformatics workflows using Hydra

Similarity-based workflow clustering

Using Domain-Specific Data to Enhance Scientific Workflow Steering Queries

Contact Info

Product

Resources

About