Skewed distributions in semi-stream joins: How much can caching help?

Naeem, M. Asif; Dobbie, Gillian; Lutteroth, Christof; Weber, Gerald

doi:10.1016/j.is.2016.09.007

Cited by 5 publications

(8 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1) Requirements/challenges for real-time stream processing for real-time DWH Following requirements and challenges for implementation of real-time stream processing for real-time DWH were identified after exploring various studies [4], [14]- [18], [20]- [31], [34]- [36], [39]- [41], [43]- [50], [52], [59], [63]- [70], [78]:…”

Section: Assessment Of Rq3: Which Approaches/tools Have Been Repormentioning

confidence: 99%

“…Some real-world trajectory datasets have been adopted by [56] and [58]: a fleet of trucks, a city buses for experimental evaluation of proposed methodologies whereas, synthetic datasets generated using the benchmark data generator were also used during evaluation by [56]. Semi-stream join algorithms developed by [20], [23], [43], [69], [70] were tested by using both synthetic and real-life datasets. They also analyzed memory and time requirements.…”

Section: Assessment Of Rq4: What Evidence Have Been Reported Whilementioning

confidence: 99%

“…In addition, to strengthen the evidence of their approaches, various studies have been identified in this literature which derived cost models to compute data latency and resources utilization: [14], [17], [20], [23], [36], [43], [45], [47], [55], [69], [70]. Moreover, research efforts should be directed towards advancing appropriate benchmarks for evaluating different real-time stream processing applications.…”

Section: Assessment Of Rq4: What Evidence Have Been Reported Whilementioning

confidence: 99%

See 2 more Smart Citations

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Mehmood

Anees

2020

IEEE Access

View full text Add to dashboard Cite

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for real-time stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter. INDEX TERMS Real-time stream processing, big data streaming, structured/un-structured data, ETL, systematic literature review Publication source Channel References No. %age IEEE Transactions on Knowledge and Data Engineering

show abstract

Section: Assessment Of Rq3: Which Approaches/tools Have Been Repormentioning

confidence: 99%

Section: Assessment Of Rq4: What Evidence Have Been Reported Whilementioning

confidence: 99%

Section: Assessment Of Rq4: What Evidence Have Been Reported Whilementioning

confidence: 99%

See 1 more Smart Citation

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Mehmood

Anees

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Content may change prior to final publication. [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16] Hadoop no Similarity [17], [18], [19] Spark no Similarity [20], [21], [22], [23], [24], [25], [26], [27] N/A yes Equi [28] N/A yes Similarity [29] Spark yes Equi DSim-Join Spark yes Similarity…”

Section: Related Workmentioning

confidence: 99%

“…Naeem, Nguyen, and Weber [23] proposed multi-way semi-stream join methods. Naeem et al [24] presented a technique for load shedding in semistream join processing. Naeem, Weber, and Lutteroth [25] proposed a semi-stream join method for the many-to-many equi-join.…”

Section: Related Workmentioning

confidence: 99%

Semi-Stream Similarity Join Processing in a Distributed Environment

Kim

Lee

2020

IEEE Access

View full text Add to dashboard Cite

Similarity join has become very important for semi-or un-structured data processing and analysis. Although several studies have been conducted on the similarity join, little attention has been paid to a semi-stream similarity join, which is a similarity join between stream data and a large diskbased relation. In this study, we propose the first distributed solution called DSim-Join for semi-stream similarity join problem. DSim-Join minimizes the data transmission, reduces database accesses using a cache in a distributed stream processing engine, parallelizes join processing, and balances the load between parallel join threads. Experimental results obtained using real-world datasets show that DSim-Join yields significantly improved throughput compared with state-of-the-art methods, especially for large datasets. The results also show that DSim-Join is scalable and stable; it is not very sensitive to the parameters such as the micro-batch interval, checkpoint interval, and similarity threshold. INDEX TERMS semi-stream join, similarity join, distributed stream processing engine, database system, big data, distributed join processing, cache management, Spark Streaming.

show abstract

A Multi-way Semi-stream Join for a Near-Real-Time Data Warehouse

Naeem

Nguyen

Weber

2017

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Semi-stream processing, the operation of joining a stream of data with non-stream disk-based master data, is a crucial component of near real-time data warehousing. The requirements for semi-stream joins are fast, accurate processing and the ability to function well with limited memory. Currently, semi-stream algorithms presented in the literature such as MeshJoin, Semi-Stream Index Join and CacheJoin can join only one foreign key in the stream data with one table in the master data. However, it is quite likely that stream data have multiple foreign keys that need to join with multiple tables in the master data. We extend CacheJoin to form three new possibilities for multi-way semistream joins, namely Sequential, Semi-concurrent, and Concurrent joins. Initially, the new algorithms can join two foreign keys in the stream data with two tables in the master data. However, these algorithms can be easily generalized to join with any number of tables in the master data. We evaluated the performance of all three algorithms, and our results show that the semi-concurrent architecture performs best under the same scenario.

show abstract

Skewed distributions in semi-stream joins: How much can caching help?

Cited by 5 publications

References 26 publications

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Semi-Stream Similarity Join Processing in a Distributed Environment

A Multi-way Semi-stream Join for a Near-Real-Time Data Warehouse

Contact Info

Product

Resources

About