A comprehensive social media data processing and analytics architecture by using big data platforms: a case study of twitter flood-risk messages

Abstract

The main objective of the article is to propose an advanced architecture and workflow based on Apache Hadoop and Apache Spark big data platforms. The primary purpose of the presented architecture is collecting, storing, processing, and analysing intensive data from social media streams. This paper presents how the proposed architecture and data workflow can be applied to analyse Tweets with a specific flood topic. The secondary objective, trying to describe the flood alert situation by using only Tweet messages and exploring the informative potential of such data is demonstrated as well. The predictive machine learning approach based on Bayes Theorem was utilized to classify flood and no flood messages. For this study, approximately 100,000 Twitter messages were processed and analysed. Messages were related to the flooding domain and collected over a period of 5 days (14 May-18 May 2018). Spark application was developed to run data processing commands automatically and to generate the appropriate output data. Results confirmed the advantages of many well-known features of Spark and Hadoop in social media data processing. It was noted that such technologies are prepared to deal with social media data streams, but there are still challenges that one has to take into account. Based on the flood tweet analysis, it was observed that Twitter messages with some considerations are informative enough to be used to estimate general flood alert situations in particular regions. Text analysis techniques proved that Twitter messages contain valuable flood-spatial information.

Overview on Data Ingestion and Schema Matching

“…In this framework, the data collection layer employs the Kafka tool to ingest data, transferring it to the ETL layer for necessary transformations and processing before storing it in the data warehouse. Similarly, (12) used another tool called Apache Flume for data collection, representing the data ingestion phase. Another innovative framework presented by (13) uses two tools, Apache Kafka and Apache Flume, as an ingestion data layer to address the challenges of data heterogeneity and velocity.…”

“…With the appropriate data ingestion tool, one can instantly import, process and store data from various data sources. (12) A first category of tools comprises software that iteratively collects data through preconceived and industrialized tasks. Most of these tools are offered by the Apache Foundation, and can also be used to aggregate, convert and clean data before ingestion.…”

Social Media Analytics Lens: A Study on Consumer Perceived Risk to Share Information on Social Media Networks

“…Furthermore, organizations are considered to operate with ethical care, but on the other hand, they are collecting data without consent. It may create a conflict of interest and ethical violations (Podhoranyi, 2021). This study aims to fill the literature gap regarding the ethicality of social media analytical practices and how it harms consumer self-disclosure and marketing discomfort.…”