Untitled

Goderis, Antoon; Fisher, Paul; Gibson, Andrew; Tanoh, Franck; Wolstencroft, Katy; Roure, David De; Goble, Carole

doi:10.1002/cpe.v21:16

Cited by 15 publications

(2 citation statements)

References 190 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Being a meta-scheduler, it relies on the centralized Condor workload management system for scheduling. Pegasus [23] extends DAGMan in order to allow mapping of workflows jobs on distributed resources and from the description of computation tasks, it performs the necessary data transfers among sites. Pegasus aims at optimizing workflow performance and reliability by scheduling to appropriate resources, but, unlike our system, there are no QoD guarantees on continuous processing or dataflow, and no data sharing.…”

Section: Related Workmentioning

confidence: 99%

Incremental dataflow execution, resource efficiency and probabilistic guarantees with Fuzzy Boolean nets

Esteves

Silva

Carvalho

et al. 2015

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Incremental dataflow execution, resource efficiency and probabilistic guarantees with Fuzzy Boolean nets

Esteves

Silva

Carvalho

et al. 2015

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

“…Pegasus [16] is a long running project that extends DAGMan in order to allow mapping of workflows of jobs to remote clusters, and cloud computing infrastructures. It maps jobs on distributed resources and from the description of computation tasks, it performs necessary data transfers (required files) among sites.…”

Section: Workflow Systemsmentioning

confidence: 99%

Fluχ: a quality-driven dataflow model for data intensive computing

Esteves

Silva

Veiga

2013

J Internet Serv Appl

View full text Add to dashboard Cite

Today, there is a growing need for organizations to continuously analyze and process large waves of incoming data from the Internet. Such data processing schemes are often governed by complex dataflow systems, which are deployed atop highly-scalable infrastructures that need to manage data efficiently in order to enhance performance and alleviate costs. Current workflow management systems enforce strict temporal synchronization among the various processing steps; however, this is not the most desirable functioning in a large number of scenarios. For example, considering dataflows that continuously analyze data upon the insertion/update of new entries in a data store, it would be wise to assess the level of modifications in data, before the trigger of the dataflow, that would minimize the number of executions (processing steps), reducing overhead and augmenting performance, while maintaining the dataflow processing results within certain coverage and freshness limit. Towards this end, we introduce the notion of Quality-of-Data (QoD), which describes the level of modifications necessary on a data store to trigger processing steps, and thus conveying in the level of performance specified through data requirements. Also, this notion can be specially beneficial in cloud computing, where a dataflow computing service (SaaS) may provide certain QoD levels for different budgets. In this article we propose Fluχ, a novel dataflow model, with framework and programming library support, for orchestrating data-based processing steps, over a NoSQL data store, whose triggering is based on the evaluation and dynamic enforcement of QoD constraints that are defined (and possibly adjusted automatically) for different sets of data. With Fluχ we demonstrate how dataflows can be leveraged to respond to quality boundaries that bring controlled and augmented performance, rationalization of resources, and task prioritization.

show abstract