Real time clustering of tweets using adaptive PSO technique and MapReduce

Chunne, Akhilesh P.; Chandrasekhar, Uddagiri; Malhotra, Chetan P.

doi:10.1109/gcct.2015.7342704

Cited by 9 publications

(2 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experiments show that parallel QPSO is superior to the serial version in terms of search capability and solution quality. There are also some related to specific application issues in the works, such as large-scale network intrusion detection systems [17,18], real-time clustering of Tweets [19], and minimizing thermal residual forces in ceramic matrix composites [20]. Parallel PSOs based on MapReduce perform well in reducing time and coping with large amounts of data.…”

Section: Related Workmentioning

confidence: 99%

Parallel Particle Swarm Optimization Based on Spark for Academic Paper Co-Authorship Prediction

et al. 2021

View full text Add to dashboard Cite

The particle swarm optimization (PSO) algorithm has been widely used in various optimization problems. Although PSO has been successful in many fields, solving optimization problems in big data applications often requires processing of massive amounts of data, which cannot be handled by traditional PSO on a single machine. There have been several parallel PSO based on Spark, however they are almost proposed for solving numerical optimization problems, and few for big data optimization problems. In this paper, we propose a new Spark-based parallel PSO algorithm to predict the co-authorship of academic papers, which we formulate as an optimization problem from massive academic data. Experimental results show that the proposed parallel PSO can achieve good prediction accuracy.

show abstract

Section: Related Workmentioning

confidence: 99%

Parallel Particle Swarm Optimization Based on Spark for Academic Paper Co-Authorship Prediction

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In the work of Chunne et al, 20 the authors used a PSO clustering algorithm running on the MapReduce platform to cluster streaming twitter data. Twitter data was pre-processed through three phases: tokenizing, stemming, and filtering by removing stop words, before applying the PSO clustering algorithm.…”

Section: Related Workmentioning

confidence: 99%

Parallel particle swarm optimization classification algorithm variant implemented with Apache Spark

Al‐Sawwa

Ludwig

2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary With the rapid development of technologies such as the internet, the amount of data that are collected or generated in many areas such as in the agricultural, biomedical, and finance sectors poses challenges to the scientific community because of the volume and complexity of the data. Furthermore, the need of analysis tools that extract useful information for decision support has been receiving more attention in order for researchers to find a scalable solution to traditional algorithms. In this paper, we proposed a scalable design and implementation of a particle swarm optimization classification (SCPSO) approach that is based on the Apache Spark framework. The main idea of the SCPSO algorithm is to find the optimal centroid for each target label using particle swarm optimization and then assign unlabeled data points to the closest centroid. Two variants of SCPSO, SCPSO‐F1 and SCPSO‐F2, were proposed based on different fitness functions, which were tested on real data sets in order to evaluate their scalability and performance. The experimental results revealed that SCPSO‐F1 and SCPSO‐F2 scale very well with increasing data set sizes and the speedup of SCPSO‐F2 is almost identical to the linear speedup while the speedup of SCPSO‐F1 is very close to the linear speedup. Thus, SCPSO‐F1 and SCPSO‐F2 can be efficiently parallelized using the Apache Spark framework.

show abstract

HTwitt: a hadoop-based platform for analysis and visualization of streaming Twitter data

Demirbaga

2021

Neural Comput & Applic

View full text Add to dashboard Cite

Twitter produces a massive amount of data due to its popularity that is one of the reasons underlying big data problems. One of those problems is the classification of tweets due to use of sophisticated and complex language, which makes the current tools insufficient. We present our framework HTwitt, built on top of the Hadoop ecosystem, which consists of a MapReduce algorithm and a set of machine learning techniques embedded within a big data analytics platform to efficiently address the following problems: (1) traditional data processing techniques are inadequate to handle big data; (2) data preprocessing needs substantial manual effort; (3) domain knowledge is required before the classification; (4) semantic explanation is ignored. In this work, these challenges are overcome by using different algorithms combined with a Naïve Bayes classifier to ensure reliability and highly precise recommendations in virtualization and cloud environments. These features make HTwitt different from others in terms of having an effective and practical design for text classification in big data analytics. The main contribution of the paper is to propose a framework for building landslide early warning systems by pinpointing useful tweets and visualizing them along with the processed information. We demonstrate the results of the experiments which quantify the levels of overfitting in the training stage of the model using different sizes of real-world datasets in machine learning phases. Our results demonstrate that the proposed system provides high-quality results with a score of nearly 95% and meets the requirement of a Hadoop-based classification system.

show abstract

Real time clustering of tweets using adaptive PSO technique and MapReduce

Cited by 9 publications

References 11 publications

Parallel Particle Swarm Optimization Based on Spark for Academic Paper Co-Authorship Prediction

Parallel Particle Swarm Optimization Based on Spark for Academic Paper Co-Authorship Prediction

Parallel particle swarm optimization classification algorithm variant implemented with Apache Spark

HTwitt: a hadoop-based platform for analysis and visualization of streaming Twitter data

Contact Info

Product

Resources

About