Fault-Tolerant Distributed Stream Processing System

Gorawski, Marcin; Marks, Paweł

doi:10.1109/dexa.2006.61

Cited by 5 publications

(2 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently there have been many studies on distributed data stream query processing [5,17,20,23,25,31,32,38,42,43,52,53]. In these studies, either of the following two types of distributed data stream models is considered: single (central) processing site model and multiple (distributed) processing site model.…”

Section: Distributed Data Stream Query Processingmentioning

confidence: 99%

Distributed stream join query processing with semijoins

Tran

Lee

2010

Distrib Parallel Databases

View full text Add to dashboard Cite

This paper addresses the distributed stream processing of window-based multi-way join queries considering the semijoin as a key join operator. In distributed stream processing, data streams arriving at remote sites need to be shipped to the processing site for query execution. This typically introduces high communication overhead. Our observation is that semijoin, effective in reducing communication overhead in distributed database query processing, can be also effective in distributed stream query processing. The challenge, however, lies in the streaming nature of the tuples, as it requires continuous and incremental processing of an unbounded sequence of tuples instead of one-time processing of a set of stored tuples. This paper describes our comprehensive work done to address the challenge. Specifically, we first propose a distributed stream join processing model that handles the issue of network delays introduced from the shipment of data streams, and allows for efficient batch processing. Then, based on the model, we propose join algorithms in a multi-way join case: first, one-way join algorithms for different combinations of join placement and join method and, then, multi-way join algorithms assuming linear join ordering. Regarding the join method, two distributed join methods are introduced:(1) simple join, in which full tuples are forwarded to the query processing site and (2) semijoin-based join, in which partial tuples are forwarded. A semijoin-based join can be executed with different possible semijoin strategies which incur different communication overheads. We present a complete set of join algorithms considering all possible semijoin strategies, and propose an optimization algorithm. The join algorithms are executed continuously in an incremental manner as tuples arrive, and never Communicated by Ling Liu. 212 Distrib Parallel Databases (2010) 27: 211-254 ship tuples redundantly. The optimization algorithm constructs an efficient multi-way join plan by using a greedy heuristic which adds to the plan one stream with the minimum join execution cost in each step. Through extensive experiments, we conduct comparative studies of the performance among the proposed one-way join algorithms and the efficiency of the generated plan between the optimization algorithm based on the greedy heuristic and the exhaustive search, respectively.

show abstract

Section: Distributed Data Stream Query Processingmentioning

confidence: 99%

Distributed stream join query processing with semijoins

Tran

Lee

2010

Distrib Parallel Databases

View full text Add to dashboard Cite

show abstract

“…They also show how to assure a DW consistency and avoid refreshing anomalies based the state-of-theart ETL and database technologies. Gorawski and Marks provide a foundation for building a fault-tolerant data stream ETL architecture (Gorawski & Marks, 2006;Gorawski & Marks, 2008). The foundation is based on the impact analysis between the components of the ETL architecture and redundant component that process the same data streams in parallel.…”

Section: Data Stream Processingmentioning

confidence: 99%

On-Demand ELT Architecture for Right-Time BI

Waas¹,

Wrembel

Freudenreich

et al. 2013

International Journal of Data Warehousing and Mining

View full text Add to dashboard Cite

In a typical BI infrastructure, data, extracted from operational data sources, is transformed, cleansed, and loaded into a data warehouse by a periodic ETL process, typically executed on a nightly basis, i.e., a full day’s worth of data is processed and loaded during off-hours. However, it is desirable to have fresher data for business insights at near real-time. To this end, the authors propose to leverage a data warehouse’s capability to directly import raw, unprocessed records and defer the transformation and data cleaning until needed by pending reports. At that time, the database’s own processing mechanisms can be deployed to process the data on-demand. Event-processing capabilities are seamlessly woven into our proposed architecture. Besides outlining an overall architecture, the authors also developed a roadmap for implementing a complete prototype using conventional database technology in the form of hierarchical materialized views.

show abstract

Temporal Costs of Computing Unit Redundancy in Steady and Transient State

Stój

Kwiecień

2017

Towards a Synergistic Combination of Research and Practice in Software Engineering

View full text Add to dashboard Cite

Fault-Tolerant Distributed Stream Processing System

Cited by 5 publications

References 4 publications

Distributed stream join query processing with semijoins

Distributed stream join query processing with semijoins

On-Demand ELT Architecture for Right-Time BI

Temporal Costs of Computing Unit Redundancy in Steady and Transient State

Contact Info

Product

Resources

About