An Anomaly Data Mining Method for Mass Sensor Networks Using Improved PSO Algorithm Based on Spark Parallel Framework

Yuan, Jingzhen

doi:10.1007/s10723-020-09505-3

Cited by 17 publications

(11 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhang et al [1] introduced incremental CFS algorithm for clustering the big data in industrial IoT. Subsequently, Yuan et al [16] introduced spark based improved particle swarm optimizer for the anomaly data mining in mass sensor networks. However, in the last few years, the data mining challenges of IoT based problems have been research trend [3].…”

Section: A Big Data In Cognitive Iot: Opportunities and Challengesmentioning

confidence: 99%

See 1 more Smart Citation

A Parallel Military-Dog-Based Algorithm for Clustering Big Data in Cognitive Industrial Internet of Things

Tripathi

Sharma

Bala³

et al. 2021

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

With the advancement of wireless communication, internet of things, and big data, high performance data analytic tools and algorithms are required. Data clustering, a promising analytic technique is widely used to solve the IoT and big data based problems, since it does not require labeled datasets. Recently, meta-heuristic algorithms have been efficiently used to solve various clustering problems. However, to handle big data sets produced from IoT devices, these algorithm fail to respond within desired time due to high computation cost. This paper presents a new meta-heuristic based clustering method to solve the big data problems by leveraging the strength of MapReduce. The proposed methods leverages the searching potential of military dog squad to find the optimal centroids and MapReduce architecture to handle the big data sets. The optimization efficacy the proposed method is validated against 17 benchmark functions and the results are compared with 5 other recent algorithms namely, bat, particle swarm optimization, artificial bee colony, multiverse optimization, and whale optimization algorithm. Further, a parallel version of the proposed method is introduced using MapReduce (MR-MDBO) for clustering the big datasets produced from industrial IoT. Moreover, the performance of MR-MDBO is studied on 2 benchmark UCI datasets and 3 real IoT based datasets produced from industry. The F-measure and computation time of the MR-MDBO is compared with the 6 other state-of-the-art methods. The experimental results witness that the proposed MR-MDBO based clustering outperforms the other considered algorithms in terms of clustering accuracy and computation times.

show abstract

Section: A Big Data In Cognitive Iot: Opportunities and Challengesmentioning

confidence: 99%

“…4) The performance of MR-MDBO is validated on 2 benchmark UCI datsets and 3 real datasets produced from industrial IoT. The results are compared against six MapReduce based state-of-the-art clustering algorithms namely, MR-KPSO [11], improved PSO [16], DFBP-KBA [17], MR-ABC [18], and MR-EGWO [19]. Rest of the paper is presented as follows.…”

Section: Introductionmentioning

confidence: 99%

A Parallel Military-Dog-Based Algorithm for Clustering Big Data in Cognitive Industrial Internet of Things

Tripathi

Sharma

Bala³

et al. 2021

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

show abstract

“…The virtualization, distributed storage, and parallel computing technologies in cloud computing provide new ideas for constructing computing platforms for data centers’ power equipment condition monitoring. It is possible to integrate the existing basic computing facilities of electric power enterprises and provide reliable, stable, and powerful storage and computing capacity support which is beneficial to monitor the online power equipment over a wide range [ 37 , 38 ].…”

Section: Introductionmentioning

confidence: 99%

Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations

AL-Jumaili

Muniyandi

Hasan

et al. 2023

Sensors

View full text Add to dashboard Cite

Traditional parallel computing for power management systems has prime challenges such as execution time, computational complexity, and efficiency like process time and delays in power system condition monitoring, particularly consumer power consumption, weather data, and power generation for detecting and predicting data mining in the centralized parallel processing and diagnosis. Due to these constraints, data management has become a critical research consideration and bottleneck. To cope with these constraints, cloud computing-based methodologies have been introduced for managing data efficiently in power management systems. This paper reviews the concept of cloud computing architecture that can meet the multi-level real-time requirements to improve monitoring and performance which is designed for different application scenarios for power system monitoring. Then, cloud computing solutions are discussed under the background of big data, and emerging parallel programming models such as Hadoop, Spark, and Storm are briefly described to analyze the advancement, constraints, and innovations. The key performance metrics of cloud computing applications such as core data sampling, modeling, and analyzing the competitiveness of big data was modeled by applying related hypotheses. Finally, it introduces a new design concept with cloud computing and eventually some recommendations focusing on cloud computing infrastructure, and methods for managing real-time big data in the power management system that solve the data mining challenges.

show abstract

“…For example, by clustering movie names, tweeting locations, and movie reviews, you can know which movies are the most popular in Tokyo, Shanghai, and London. e collection of diverse data may be the real value of social networking sites now, and they may be able to squeeze out advertising as a major source of revenue for social networking sites [4]. However, the new generation of big data computing method framework and programming model is running in parallel, and the traditional data mining algorithm based on the serial mode of a single machine is difficult to apply directly to the distributed framework; therefore, the parallelization research of data mining algorithm in the distributed computing framework has also become a major hot spot in the field of data mining research.…”

Section: Introductionmentioning

confidence: 99%

Clustering Algorithm for Big Datasets with Mixed Attribute Features under Spark

Wang¹

2022

Mathematical Problems in Engineering

View full text Add to dashboard Cite

This paper presents an in-depth study and analysis of large datasets of mixed and attribute features under Spark using a large dataset clustering algorithm. The classical algorithm K-means based on division and the density-based clustering algorithm DPC, which has become more popular in recent years, are selected as the research objects of this paper. Secondly, the original K-means algorithm is improved by combining holdout validation and K-means++ method to address the shortcomings of the K-means algorithm that the number of class clusters K needs to be set in advance and the initial class cluster centers are chosen randomly, which leads to unstable iterations and slow convergence of clustering results. The similarity matrix will be continuously updated during the iterative process. It mainly refers to the process of dividing objects into multiple classes according to the degree of similarity between objects. After the division, the objects within the class are like each other, while the objects between the classes are different from each other. The comparison experiments of the improved algorithm before and after the MovieLens dataset are conducted to verify that the new algorithm has better performance in terms of clustering accuracy and efficiency. Again, to address the drawback that the clustering results in the DPC algorithm rely heavily on the subjective selection of the truncation distance parameter cd, and it is difficult to handle datasets with complex distribution and large density variation, the algorithm can generate the optimal cd adaptively by combining K-nearest neighbors and introducing the distance comparison quantity, which has a better performance by considering the overall and local distribution of the data. The feasibility of the improved method is verified by validating the algorithm with artificial datasets and UCI datasets as well as separation tests. Finally, the parallelized design and implementation of the improved K-means algorithm and CDPC-KNN algorithm are completed by building a Spark clustering environment, and the parallelized algorithm is verified to have much better data processing capability and be more adaptable to the clustering analysis of large-scale data by comparing algorithm string parallelism experiments.

show abstract

An Anomaly Data Mining Method for Mass Sensor Networks Using Improved PSO Algorithm Based on Spark Parallel Framework

Cited by 17 publications

References 13 publications

A Parallel Military-Dog-Based Algorithm for Clustering Big Data in Cognitive Industrial Internet of Things

A Parallel Military-Dog-Based Algorithm for Clustering Big Data in Cognitive Industrial Internet of Things

Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations

Clustering Algorithm for Big Datasets with Mixed Attribute Features under Spark

Contact Info

Product

Resources

About