Real-time stream processing for Big Data

Wingerath, Wolfram; Gessert, Felix; Friedrich, Steffen; Ritter, Norbert

doi:10.1515/itit-2016-0002

Cited by 35 publications

(33 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In contrast to Storm, Trident API provides a stronger ordering guarantee, exactly-once processing semantics, works in micro-batches and introduces batch size as a parameter to increase throughput at the cost of latency. However, their topologies are not suitable for implementing iterative algorithms since they are directed acyclic graphs (DAGs) [57]. The architecture of Storm Trident can be found on [57].…”

Section: Apache Stormmentioning

confidence: 99%

“…However, their topologies are not suitable for implementing iterative algorithms since they are directed acyclic graphs (DAGs) [57]. The architecture of Storm Trident can be found on [57].…”

Section: Apache Stormmentioning

confidence: 99%

“…This processing framework was created to solve various kinds of stream processing requirements such as efficient use of resources and at scale, handle failures gracefully, and scalability [58]. It provides at-least-once processing semantics and once-at-a-time processing model [57]. It uses Apache Kafka to provide fault tolerance, buffering and state storage [46], and Hadoop YARN for distributed resource allocation and scheduling [56].…”

Section: Apache Samzamentioning

confidence: 99%

See 2 more Smart Citations

Review of Big Data and Processing Frameworks for Disaster Response Applications

Cumbane

Gidófalvi

2019

IJGI

View full text Add to dashboard Cite

Natural hazards result in devastating losses in human life, environmental assets and personal, and regional and national economies. The availability of different big data such as satellite imageries, Global Positioning System (GPS) traces, mobile Call Detail Records (CDRs), social media posts, etc., in conjunction with advances in data analytic techniques (e.g., data mining and big data processing, machine learning and deep learning) can facilitate the extraction of geospatial information that is critical for rapid and effective disaster response. However, disaster response systems development usually requires the integration of data from different sources (streaming data sources and data sources at rest) with different characteristics and types, which consequently have different processing needs. Deciding which processing framework to use for a specific big data to perform a given task is usually a challenge for researchers from the disaster management field. Therefore, this paper contributes in four aspects. Firstly, potential big data sources are described and characterized. Secondly, the big data processing frameworks are characterized and grouped based on the sources of data they handle. Then, a short description of each big data processing framework is provided and a comparison of processing frameworks in each group is carried out considering the main aspects such as computing cluster architecture, data flow, data processing model, fault-tolerance, scalability, latency, back-pressure mechanism, programming languages, and support for machine learning libraries, which are related to specific processing needs. Finally, a link between big data and processing frameworks is established, based on the processing provisioning for essential tasks in the response phase of disaster management.

show abstract

Section: Apache Stormmentioning

confidence: 99%

Section: Apache Stormmentioning

confidence: 99%

Section: Apache Samzamentioning

confidence: 99%

See 1 more Smart Citation

Review of Big Data and Processing Frameworks for Disaster Response Applications

Cumbane

Gidófalvi

2019

IJGI

View full text Add to dashboard Cite

show abstract

“…, [24] allows consuming live data; it divides the stream in mini batch into time periods equal to batch interval. After every batch it produces a DStream (see Fig.…”

Section: Spark Streaming Library [23]mentioning

confidence: 99%

Real-Time Analysis of Students’ Activities on an E-Learning Platform based on Apache Spark

Chaffai¹,

Hassouni²,

Anoun³

2017

ijacsa

View full text Add to dashboard Cite

Abstract-Real time analytics is the capacity to extract valuables insights from data that comes continuously from activities on the web or network sensors. It is largely used in web based business to drive decisions based on user's experiences, such dynamic pricing and personalized advertising. Many universities have adopted web based learning in their learning process. They use data-mining techniques to better understand students' behavior, and most of the tools developed are based on historical and stored data, and do not allow real time reactivity. Online activities of learners generate at high speed a huge amount of data in form of users' interactions which have all characteristics to be considered as Big data. Deal with volume and velocity of these data in order to inform and enable decisions-makers to act at right time lead us to use new methods to capture E-Learning data, and process it in real time.This paper focuses on the design and implementation of modern and hybrid real time data pipeline architecture using Apache Flume to collect data, Apache Spark as an unified engine computation for performing analytics on students' activities data and Apache Hive as a data warehouse for storing the processed data and for use by various reporting tools. To conceive this platform we conduct an experiment on Moodle database source.

show abstract

“…A modified framework is designed to perform Parallel Processing of Big data than the Hadoop is called Spark. It permits both batch and stream Processing while Hadoop mostly for batch processing [10]. Even though Spark uses Hadoop Distributed File System (HDFS) in some circumstances Spark runs hundred times faster than Hadoop.…”

Section: Introductionmentioning

confidence: 99%

A Novel Frequent Pattern Mining Technique for Prediction of User Behavior on Web Stream Data

Dhanalakshmi¹

2019

ISI

View full text Add to dashboard Cite

In recent years, as the size of the online databases increases, content in the web pages also increases, then the human behavior towards the online content has become a major issue for decision making. Hence, the need for extracting knowledge from high dimensional online content using automated techniques also increased. To address the decision making issues, a novel dynamic model is required to discover the required knowledge from these databases. In this paper, a novel filter based user navigation pattern mining model is designed and implemented on the large online steaming databases. Experimental results proved that the present filtered based frequent pattern mining model efficiently predicts the user navigation patterns with high accuracy and less runtime.

show abstract

Real-time stream processing for Big Data

Cited by 35 publications

References 6 publications

Review of Big Data and Processing Frameworks for Disaster Response Applications

Review of Big Data and Processing Frameworks for Disaster Response Applications

Real-Time Analysis of Students’ Activities on an E-Learning Platform based on Apache Spark

A Novel Frequent Pattern Mining Technique for Prediction of User Behavior on Web Stream Data

Contact Info

Product

Resources

About