Big Data Processing: Batch-based processing and stream-based processing

Benjelloun, Sarah; Aissi, Mohamed El Mehdi El; Loukili, Yassine; Lakhrissi, Younes; Ali, Safae Elhaj Ben; Chougrad, Hiba; Boushaki, Abdessamad El

doi:10.1109/icds50568.2020.9268684

Cited by 13 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are challenges/dimensions of big data called ‘the 4V’s of Big Data’: Volume, Velocity, Variety and Veracity. Meanwhile, there are two types of big data processing: batch-based processing and stream-based processing [33] . In batch-based processing, each data block is processed sequentially one by one in a period of time.…”

Section: Methodsmentioning

confidence: 99%

Big data directed acyclic graph model for real-time COVID-19 twitter stream detection

Amen

Faiz²,

Toan

2022

Pattern Recognition

View full text Add to dashboard Cite

Every day, large-scale data are continuously generated on social media as streams, such as Twitter, which inform us about all events around the world in real-time. Notably, Twitter is one of the effective platforms to update countries leaders and scientists during the coronavirus (COVID-19) pandemic. Other people have also used this platform to post their concerns about the spread of this virus and a rapid increase of death cases globally. The aim of this work is to detect anomalous events associated with COVID-19 from Twitter. To this end, we propose a distributed Directed Acyclic Graph topology framework to aggregate and process large-scale real-time tweets related to COVID-19. The core of our system is a novel lightweight algorithm that can automatically detect anomaly events. In addition, our system can also identify, cluster, and visualize important keywords in tweets. On 18 August 2020, our model detected the highest anomaly since many tweets mentioned the casualties’ updates and the debates on the pandemic that day. We obtained the three most commonly listed terms on Twitter: “covid”, “death”, and “Trump” (21566, 11779, and 4761 occurrences, respectively), with the highest TF-IDF score for these terms: “people” (0.63637), “school” (0.5921407) and “virus” (0.57385). From our clustering result, the word “death”, “corona”, and “case” are grouped into one cluster, where the word “pandemic”, “school”, and “president” are grouped as another cluster. These terms were located near each other on vector space so that they were clustered, indicating people’s most concerned topics on Twitter.

show abstract

Section: Methodsmentioning

confidence: 99%

Big data directed acyclic graph model for real-time COVID-19 twitter stream detection

Amen

Faiz²,

Toan

2022

Pattern Recognition

View full text Add to dashboard Cite

show abstract

“…Big Data processing is the process of dealing with massive amounts of data and converting it from its raw form into a useful approach and a more understandable manner (Benjelloun et al, 2020). This section highlights the most powerful frameworks used to manage massive amounts of quickly generated data.…”

Section: Big Data Processing Frameworkmentioning

confidence: 99%

“…Also, Yet Another Resource Negotiator (YARN), was released in 2012 by Yahoo! and Hortonworks (Khalid and Yousaf, 2021;Benjelloun et al, 2020;and Cumbane and Gidófalvi, 2019). The Hadoop ecosystem also includes the Hadoop kernel, as well as other components: HBase, Apache Hive, Oozie, Zookeeper, Pig, etc.…”

Section: Hadoop Frameworkmentioning

confidence: 99%

“…Stream processing is utilized when data has to be analyzed as quickly as it comes, whether it's social data or machine data from IoT systems. The goal of using stream-based processing is to achieve: low latency, and realtime reaction to a new event (Benjelloun et al, 2020). In short, streaming data processing indicates that the data will be examined and activities will be made on the data as soon as possible, usually in near real-time.…”

Section: Streaming Processingmentioning

confidence: 99%

See 1 more Smart Citation

Big Data Processing Frameworks for Handling Huge Data Efficiencies and Challenges: A Survey

Al-Barznji¹

2022

IJDSBDA

View full text Add to dashboard Cite

The increasing expansion of digital data collected from many sources renders traditional storage, processing, and analysis methods obsolete. For these restrictions, new technologies for processing and storing very massive datasets have been developed. Big data processing is required to extract relevant information from it. Transforming data into information and knowledge is what processing implies. Big data processing is the process of dealing with massive amounts of data and changing it from its raw form into useable information in a more understandable manner. As a result, numerous big data processing execution frameworks have emerged, but determining and selecting the appropriate framework for processing your big data applications is a significant challenge. Therefore, this paper investigates the possible influence of big data challenges and discusses in depth the most well-known approaches to big data processing, which are divided into five classes: batch processing, streaming processing, realtime processing, interactive processing, and hybrid processing, as well as the variety of the most popular frameworks associated with them such as Apache Hadood, Dryad, Samza, IBM Infosphere, Storm, Amazon Kinesis, Drill, Impala, Flink, and Spark. Furthermore, this study presents a comparison among the several features of the frameworks by highlighting their drawbacks and strengths. Thus, it can be used as a guideline for picking the best application framework in IT analytics and will help busi-ness users make faster decisions.

show abstract

“…When the data have previously been saved over time, this type of processing is performed. In essence, batch processing manages beginning and ending jobs [11,13].…”

Section: Batch Processing and Big Datamentioning

confidence: 99%

Digital Transformation: After the Big Data, What Next?

Udofia¹,

Buduka²,

Akpabio

et al. 2020

Day 1 Tue, August 11, 2020

View full text Add to dashboard Cite

Our definition of ‘The Big Data’ is the ability of having all the data set at the point of decision-making. Several documented research shows that less than 5% of all the data acquired in the oil and gas industry are actually utilised. Despite these challenges most operating assets, still keep acquiring more data and creating data-load that are never utilised or put into use. This paper aims to demonstrate the reason(s) why we have this data-load and instigate a solution path in managing this challenge for value realisation. Data is a corporate asset in any organisation but the default ‘silo’ work culture in most operating asset makes it difficult to get the relevant insight from these data. Organisation must be deliberate in creating the right data architecture from the data process domain through into the distributed control system, into the data historian, then within the data platform and the visualization realm and finally into the workflow domain. It is at the workflow domain that we get the ‘game-changer’ otherwise called value. This is the differentiating factor in delivering the ‘Big Data’ into the space of benefit realisation. This was what was achieved in a field in Niger Delta and has been shared compared to global best practice. The documented result from this case study showed that any data architecture should be framed to address asset business need, right from the data process domain function through to the data historian into the workflow level where the business objective is addressed. Data must be application-centric accounting for scalability, accessibility, security and business performance. Assets need to be clear on why they are acquiring data and not just continuous acquisition of more data with the resultant effect of data-load without answering the question; what next after data acquisition? This paper provides the answer of what next after data acquisition and serves as a quick reference tool for similar challenge elsewhere, not limited to the oil and gas industry.

show abstract

Big Data Processing: Batch-based processing and stream-based processing

Cited by 13 publications

References 24 publications

Big data directed acyclic graph model for real-time COVID-19 twitter stream detection

Big data directed acyclic graph model for real-time COVID-19 twitter stream detection

Big Data Processing Frameworks for Handling Huge Data Efficiencies and Challenges: A Survey

Digital Transformation: After the Big Data, What Next?

Contact Info

Product

Resources

About