A spark‐based big data analysis framework for real‐time sentiment prediction on streaming data

Abstract

Summary There are many data sources that produce large volumes of data. The Big Data nature requires new distributed processing approaches to extract the valuable information. Real‐time sentiment analysis is one of the most demanding research areas that requires powerful Big Data analytics tools such as Spark. Prior literature survey work has shown that, though there are many conventional sentiment analysis researches, there are only few works realizing sentiment analysis in real time. One major point that affects the quality of real‐time sentiment analysis is the confidence of the generated data. In more clear terms, it is a valuable research question to determine whether the owner that generates sentiment is genuine or not. Since data generated by fake personalities may decrease accuracy of the outcome, a smart/intelligent service that can identify the source of data is one of the key points in the analysis. In this context, we include a fake account detection service to the proposed framework. Both sentiment analysis and fake account detection systems are trained and tested using Naïve Bayes model from Apache Spark's machine learning library. The developed system consists of four integrated software components, ie, (i) machine learning and streaming service for sentiment prediction, (ii) a Twitter streaming service to retrieve tweets, (iii) a Twitter fake account detection service to assess the owner of the retrieved tweet, and (iv) a real‐time reporting and dashboard component to visualize the results of sentiment analysis. The sentiment classification performances of the system for offline and real‐time modes are 86.77% and 80.93%, respectively.

Twitter Sentiment Analysis Approaches: A Survey

“…A number of studies have combined two or more approaches for TSA such as combining machine-learning and lexicon-based approaches [3]. The work in [26] proposed a hybrid method by discussing a real-time sentiment analysis using Apache Spark's machine learning library, Hadoop distributed file system and streaming engine for sentiment prediction. The sentiment classification performance of the proposed system for offline and real-time modes were 86.77% and 80.93%, respectively.…”

Sentiment Drift Detection and Analysis in Real Time Twitter Data Streams

“…Kılınç [23] has claimed that data produced by fake personalities affects the sentiment analysis model. In order to overcome this, a spark based big data framework, which incorporates a sentiment analysis method and a fake detection method has been proposed.…”

Simulating real-time tweet sentiment analysis by different machine learning methods based on spark

“…There are also some papers focusing on real-time sentiment analysis. Kilinc demonstrates that a considerable challenge of real-time sentiment analysis is the uncertainty of the reliability since there exist some fake accounts due to unethical reasons, he builds a spark-based real-time sentiment prediction framework that detects the authenticity of accounts before inputting data [7]. Moreover, SVM shows an accuracy of around 85% on real-time sentiment analysis of Amazon product reviews, there are four main steps in processing data, which are tokenization, removing stop words, POS tagging, and stemming [8].…”