Streamflow Programming Model for Data Streaming in Scientific Workflows

Herath, Chathura; Plale, Beth

doi:10.1109/ccgrid.2010.116

Cited by 16 publications

(13 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [11], authors propose a framework to compensate for the impedance mismatch between scientific workflows and continuous data streams. They also propose workflow semantics to incorporate stream in scientific workflows.…”

Section: Related Workmentioning

confidence: 99%

Chemistry-Inspired Adaptive Stream Processing

2015

View full text Add to dashboard Cite

Abstract. Stream processing engines have appeared as the next generation of data processing systems, facing the needs for low-delay processing. While these systems have been widely studied recently, their ability to adapt their processing logics at run time upon the detection of some events calling for adaptation is still an open issue. Chemistry-inspired models of computation have been shown to ease the specification of adaptive systems. In this paper, we argue that a higherorder chemical model can be used to specify such an adaptive SPE in a natural way. We also show how such programming abstractions can get enacted in a decentralised environment.

show abstract

Section: Related Workmentioning

confidence: 99%

Chemistry-Inspired Adaptive Stream Processing

2015

View full text Add to dashboard Cite

show abstract

“…Of course, a naïve implementation involves computing the cumulative sum by simply adding each element of the input to the previous element of the output. Such an approach would have a runtime of N. However, a more efficient approach has existed since the development of APL if not for longer 16 . To ensure that the reader has some intuition as to how this could be possible, the key idea is to exploit the partial sums that are calculated in an adder tree and to express each element of the cumulative sum as a sum of these (efficiently calculated) partial sums.…”

Section: Cumulative Summentioning

confidence: 99%

“…These models include MapReduce [15], Stream Processing [11,12,16] and Query-based techniques [17,18]. Here, we focus on one such programming model, MapReduce.…”

Section: Big Data Frameworkmentioning

confidence: 99%

“…15 Note that the cumulative sum is sometimes referred to as a prefix sum: there is no difference between a prefix sum and a cumulative sum. 16 APL describes an approach to calculating a sum, maximum or minimum as reduction operations. The approach to calculating a cumulative sum is described as a scan operation and can be used to calculate, for example, cumulative maximums and minimums.…”

Section: Endnotesmentioning

confidence: 99%

See 1 more Smart Citation

MapReduce particle filtering with exact resampling and deterministic runtime

Thiyagalingam

Kekempanos

Maskell

2017

EURASIP J. Adv. Signal Process.

View full text Add to dashboard Cite

Particle filtering is a numerical Bayesian technique that has great potential for solving sequential estimation problems involving non-linear and non-Gaussian models. Since the estimation accuracy achieved by particle filters improves as the number of particles increases, it is natural to consider as many particles as possible. MapReduce is a generic programming model that makes it possible to scale a wide variety of algorithms to Big data. However, despite the application of particle filters across many domains, little attention has been devoted to implementing particle filters using MapReduce. In this paper, we describe an implementation of a particle filter using MapReduce. We focus on a component that what would otherwise be a bottleneck to parallel execution, the resampling component. We devise a new implementation of this component, which requires no approximations, has O(N) spatial complexity and deterministic O (log N) 2 time complexity. Results demonstrate the utility of this new component and culminate in consideration of a particle filter with 2 24 particles being distributed across 512 processor cores.

show abstract

“…They have the total control over a dedicated optical network so that a deterministic path can be obtained with advance reservation or on demand, while our work uses a TCP/IP based shared network, such that bandwidth is more of an issue. Streamflow [40] proposes a framework that enhances a traditional scientific workflow infrastructure to enable the stream processing capability, and resource allocation is not discussed. In [41], a distributed and pipelined dataflow execution system is proposed.…”

Section: E Performance Comparisonmentioning

confidence: 99%

Dynamic Control of Data Streaming and Processing in a Virtualized Environment

Cao

Wen

Tan

2012

IEEE Trans. Automat. Sci. Eng.

View full text Add to dashboard Cite

Abstract-Performance of data streaming applications is co-determined by both networking and computing resources, and therefore they should be co-scheduled and co-allocated in an integrated and coordinated way. Dynamic control of resource scheduling and allocation is required, because unilateral redundancy in either networking or computing resources may result in the overprovision of it and the other may become a bottleneck. To avoid resource shortage as well as overprovision, in this paper, a virtualized platform is utilized to implement data streaming and processing. In this platform, fuzzy logic controllers are designed to allocate CPU resources; iterative bandwidth allocation is applied and is processing-and storage-aware to guarantee on-demand data provisioning. Experimental results show that our approach leads to higher application performance as well as higher resource utilization, compared with other resource scheduling and allocation methods.Note to Practitioners-Data streaming has become an important paradigm in many business and scientific applications, such as fraud detection in banking industry and gravitational-wave observation in astronomy. In this paper, we apply virtualization to provide better support for these applications, leveraging its advantage in resource on-demand allocation. A novel dynamic control method is proposed so that CPU and bandwidth can be co-scheduled and co-allocated since for data streaming applications these resources are tightly coupled from the performance perspective. In this control method, fuzzy logic control is applied for CPU allocation and an iterative algorithm is adopted for processing-, congestion-, and storage-aware bandwidth allocation. We use fuzzy control because it does not rely on the mathematical modeling of an object and can implement human experts' heuristic knowledge via IF-THEN rules. Therefore, it is desirable in our data streaming scenario, due to the variable coupling and heavily nonlinear nature of the system.

show abstract

Streamflow Programming Model for Data Streaming in Scientific Workflows

Cited by 16 publications

References 20 publications

Chemistry-Inspired Adaptive Stream Processing

Chemistry-Inspired Adaptive Stream Processing

MapReduce particle filtering with exact resampling and deterministic runtime

Dynamic Control of Data Streaming and Processing in a Virtualized Environment

Contact Info

Product

Resources

About