A Survey of Distributed Data Stream Processing Frameworks

Isah, Haruna; Abughofa, Tariq; Mahfuz, Sazia; Ajerla, Dharmitha; Zulkernine, Farhana; Khan, Shahzad

doi:10.1109/access.2019.2946884

Cited by 117 publications

(52 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Once the paper is found, inclusion/exclusion of that paper has been decided after reading its abstract and then other parts of paper. After having examined selected papers thoroughly we identified one more study [5], and totally added up to 74 primary studies.…”

Section: ) Selection Based On Snowballingmentioning

confidence: 99%

“…Big data analysis platforms and tools have been reviewed in a study [3] along with their applications, such as: Hadoop, GridGain, MapReduce, HPCC and Apache Storm. Whereas difficulties in selecting the right stream processing framework were identified and addressed for different use cases while developing a streaming analytics infrastructure by [5]. This study presents critical review of key features of some stream processing engines including Storm, Spark Streaming, Flink, Kafka Streams, IBM Steams.…”

Section: B Assessment Of Rq2: Which Challenges Have Been Faced Durinmentioning

confidence: 99%

“…Another recent comparative study of data stream analytics frameworks is presented by [5]. Different data stream processing engines have been evaluated in this study based on their partitioning, state management, message delivery and fault tolerance features.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Mehmood

Anees

2020

IEEE Access

View full text Add to dashboard Cite

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for real-time stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter. INDEX TERMS Real-time stream processing, big data streaming, structured/un-structured data, ETL, systematic literature review Publication source Channel References No. %age IEEE Transactions on Knowledge and Data Engineering

show abstract

Section: ) Selection Based On Snowballingmentioning

confidence: 99%

Section: B Assessment Of Rq2: Which Challenges Have Been Faced Durinmentioning

confidence: 99%

See 1 more Smart Citation

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Mehmood

Anees

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Finally, an interesting extension of our approach is to use open-source distributed stream processing systems, like Storm, Flink, or Spark Streaming [67]- [69] to carry our real-time analytics. As shown in several works in literature (e.g., [70]- [72]), the common use-case exploits a distributed messaging system, like Apache Kafka [73], that runs on the IoT devices and sends data to dedicated servers hosting the streaming analytics software.…”

Section: Computational Load and Scalabilitymentioning

confidence: 99%

pAElla: Edge AI-Based Real-Time Malware Detection in Data Centers

Libri

Bartolini

Benini

2020

IEEE Internet Things J.

View full text Add to dashboard Cite

The increasing use of Internet-of-Things (IoT) devices for monitoring a wide spectrum of applications, along with the challenges of "big data" streaming support they often require for data analysis, is nowadays pushing for an increased attention to the emerging edge computing paradigm. In particular, smart approaches to manage and analyze data directly on the network edge, are more and more investigated, and Artificial Intelligence (AI) powered edge computing is envisaged to be a promising direction. In this paper, we focus on Data Centers (DCs) and Supercomputers (SCs), where a new generation of high-resolution monitoring systems is being deployed, opening new opportunities for analysis like anomaly detection and security, but introducing new challenges for handling the vast amount of data it produces. In detail, we report on a novel lightweight and scalable approach to increase the security of DCs / SCs, that involves AI-powered edge computing on high-resolution power consumption. The method -called pAElla -targets real-time Malware Detection (MD), it runs on an out-of-band IoT-based monitoring system for DCs / SCs, and involves Power Spectral Density of power measurements, along with AutoEncoders. Results are promising, with an F1-score close to 1, and a False Alarm and Malware Miss rate close to 0%. We compare our method with State-of-the-Art MD techniques and show that, in the context of DCs / SCs, pAElla can cover a wider range of malware, significantly outperforming SoA approaches in terms of accuracy. Moreover, we propose a methodology for online training suitable for DCs / SCs in production, and release open dataset and code.

show abstract

“…The core of a DSPS is the Data Stream Processing Engine, DSPE, which enables the definition and execution of stream processing pipelines [4]. A review of the state of the art shows many alternatives for DSPEs [5] whereas the number of existing DSPSs is very limited [4].…”

Section: Introductionmentioning

confidence: 99%

Streamer

García-Rodríguez

Alshaer

Gouy-Pailler

2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

With the proliferation of continuous data generation, data stream processing has become a key topic in research. As a consequence, the need for dedicated tools to apply continuous learning in streams emerges. This paper presents STREAMER, a flexible, scalable, and cross-platform machine learning experimenter with a realistic operational stream environment and visualization capabilities. Oriented to data scientists, this framework provides a set of machine learning algorithms and an API to easily integrate new ones. In order to illustrate how STREAMER works, we show a demonstration of an unsupervised anomaly detection of electrocardiograms (ECG) tested in a streaming context.

show abstract

A Survey of Distributed Data Stream Processing Frameworks

Cited by 117 publications

References 49 publications

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

pAElla: Edge AI-Based Real-Time Malware Detection in Data Centers

Streamer

Contact Info

Product

Resources

About