Tracking Dengue on Twitter Using Hybrid Filtration-Polarity and Apache Flume

Abstract

The world health organization (WHO) terms dengue as a serious illness that impacts almost half of the world's population and carries no specific treatment. Early and accurate detection of spread in affected regions can save precious lives. Despite the severity of the disease, a few noticeable works can be found that involve sentiment analysis to mine accurate intuitions from the social media text streams. However, the massive data explosion in recent years has led to difficulties in terms of storing and processing large amounts of data, as reliable mechanisms to gather the data and suitable techniques to extract meaningful insights from the data are required. This research study proposes a sentiment analysis polarity approach for collecting data and extracting relevant information about dengue via Apache Hadoop. The method consists of two main parts: the first part collects data from social media using Apache Flume, while the second part focuses on querying and extracting relevant information via the hybrid filtration-polarity algorithm using Apache Hive. To overcome the noisy and unstructured nature of the data, the process of extracting information is characterized by pre and post-filtration phases. As a result, only with the integration of Flume and Hive with filtration and polarity analysis, can a reliable sentiment analysis technique be offered to collect and process large-scale data from the social network. We introduce how the Apache Hadoop ecosystem -Flume and Hivecan provide a sentiment analysis capability by storing and processing large amounts of data. An important finding of this paper is that developing efficient sentiment analysis applications for detecting diseases can be more reliable through the use of the Hadoop ecosystem components than through the use of normal machines.

Research on ship manning data analysis based on data warehouse

“…The platform uses flume to read the data in the message queue and write the data to HDFS. Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of data [8]. The platform sets up Flume agent, the source end connects to Kafka, the destination end connects to HDFS, and the data is transferred to HDFS through the pipeline mode.…”

Sentiment Drift Detection and Analysis in Real Time Twitter Data Streams

“…Ghani et al [11] have developed a system using Flume, to track dengue epidemic using hybrid filtration polarity. It calculates the sentiment analysis value of the twitter data to extract the useful information from the tweet content.…”