Processing Exact Results for Sliding Window Joins over Time-Sequence, Streaming Data Using a Disk Archive

Chakraborty, Abhirup; Singh, Ajit

doi:10.1109/aciids.2009.64

Cited by 2 publications

(4 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The alternative, then, is to use secondary storage as an extension of main memory. Indeed, such disk-based approach has been adopted in a number of proposals such as [6], [10] and [15]. The focus of those works, though, is on processing of join queries.…”

Section: Related Workmentioning

confidence: 98%

“…Farag and Hamad [10] propose a two-phase external-memory algorithm that joins the arriving tuples of one stream with the memory-resident data of the other streams, and postpones matching with the disk-resident portion until the stream runs out-of-space or arrival of new tuples stalls. Chakraborty and Singh [6] propose an Exact Window Join algorithm that deals with memory shortages by deferring the load during high workload, and processing the deferred load during the period of low workload. This strategy, however, results in high delays (> 5 seconds) even for moderate data input rates (450 tuples per second).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Overcoming memory limitations in high-throughput event-based applications

Mendes

Bizarro

Marques

2013

Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

The last decade has witnessed the emergence of business critical applications processing streaming data for domains as diverse as credit card fraud detection, real-time recommendation systems, call-center monitoring, ad selection, network monitoring, and more. Most of those applications need to compute hundreds or thousands of metrics continuously while coping with very high event input rates. As a consequence, large amounts of state (i.e., moving windows) need to be maintained, very often exceeding the available memory resources. Nonetheless, current event processing platforms have little or no memory management capabilities, hanging or simply crashing when memory is exhausted. In this paper we report our experience in using secondary storage for solving the performance problems of memory-constrained event processing applications. For that, we propose SlideM, a novel buffer management algorithm that exploits the access pattern of sliding windows in order to efficiently handle memory shortages. The proposed algorithm was implemented in a real stream processing engine and validated through an extensive experimental performance evaluation. Results corroborate the efficacy of the approach: the system was able to sustain very high input rates (up to 300,000 events per second) for very large windows (about 30GB) while consuming small amounts of main memory (few kilobytes).

show abstract

Section: Related Workmentioning

confidence: 98%

Section: Related Workmentioning

confidence: 99%

Overcoming memory limitations in high-throughput event-based applications

Mendes

Bizarro

Marques

2013

Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

show abstract

“…Acquisition of exact result of processing of joining streams has been focused by [8]. Authors have introduced an algorithm (EWJ) which produces correct and exact result of joining streams with limited memory.…”

Section: Application Of Data Mining In Crime Detectionmentioning

confidence: 99%

“…Not only handle reliable network but also handle unreliable network [8] How to get exact processing and sliding windows among data streams?…”

Section: Time Granularity Adjustmentmentioning

confidence: 99%

Challenges and Issues in DATA Stream: A Review

Arif¹,

Alam²,

Hussain³

2015

IJHIT

View full text Add to dashboard Cite

Data stream is a continuous, time varying, massive and infinitely ordered sequence of data elements. The streaming data are fast changing with time, it is impossible to acquire all the elements in a data stream. Therefore, each data element should be examined at most once in data streams. Memory usage for mining data stream should be limited due to the new data elements are continuously generated from the streams. It is essential to ensure that newly arrived stream should be immediately available whenever it is requested made this task much challenging and necessary for fraud detection in stream, taking out knowledge, for business improvement and other applications where data arrived in stream. This paper tries to highlight important issues and research challenges of data stream by means of a comprehensive review.

show abstract

Processing Exact Results for Sliding Window Joins over Time-Sequence, Streaming Data Using a Disk Archive

Cited by 2 publications

References 16 publications

Overcoming memory limitations in high-throughput event-based applications

Overcoming memory limitations in high-throughput event-based applications

Challenges and Issues in DATA Stream: A Review

Contact Info

Product

Resources

About