A Classification Algorithm Based on Data Clustering and Data Reduction for Intrusion Detection System over Big Data

Abstract

With the rapid development of network, Intrusion Detection System(IDS) plays a more and more important role in network applications. Many data mining algorithms are used to build IDS. However, due to the advent of big data era, massive data are generated. When dealing with large-scale data sets, most data mining algorithms suffer from a high computational burden which makes IDS much less efficient. To build an efficient IDS over big data, we propose a classification algorithm based on data clustering and data reduction. In the training stage, the training data are divided into clusters with similar size by Mini Batch K-Means algorithm, meanwhile, the center of each cluster is used as its index. Then, we select representative instances for each cluster to perform the task of data reduction and use the clusters that consist of representative instances to build a K-Nearest Neighbor(KNN) detection model. In the detection stage, we sort clusters according to the distances between the test sample and cluster indexes, and obtain k nearest clusters where we find k nearest neighbors. Experimental results show that searching neighbors by cluster indexes reduces the computational complexity significantly, and classification with reduced data of representative instances not only improves the efficiency, but also maintains high accuracy.

User value identification based on an improved consumer value segmentation algorithm

“…Considering the operation efficiency of the algorithm, Sculley (2010) proposed the MBK algorithm, in which when training the clustering centers, a subset was randomly selected to train the clustering centers, which reduces the convergence time effectively. The pseudo code of the MBK algorithm is shown in Algorithm 1 (Wang et al, 2019) (Fan, 2019). The pseudo code of the Chameleon algorithm is shown in Algorithm 2 (Zhang et al, 2021a, b).…”

“…Obviously, the KMA algorithm is not the best choice for market segmentation. Based on this, Sculley (2010) improved the KMA and proposed the mini-batch k-means (MBK) algorithm, which has advantages as respect to the learning speed and efficiency while maintaining the clustering performance similar to that of the KMA (Nam, 2019); to the best of our current knowledge, there are no relevant scholars specializing in studying the application of MBK in the field of market segmentation, while this algorithm has been widely used in biological field to detect distinct subpopulations of cells, Internet field to build an efficient intrusion detection system over big data and other fields (Wang et al. , 2019; Hicks et al.…”

“…Obviously, the KMA algorithm is not the best choice for market segmentation. Based on this, Sculley (2010) improved the KMA and proposed the mini-batch k-means (MBK) algorithm, which has advantages as respect to the learning speed and efficiency while maintaining the clustering performance similar to that of the KMA (Nam, 2019); to the best of our current knowledge, there are no relevant scholars specializing in studying the application of MBK in the field of market segmentation, while this algorithm has been widely used in biological field to detect distinct subpopulations of cells, Internet field to build an efficient intrusion detection system over big data and other fields (Wang et al, 2019;Hicks et al, 2020;Yang et al, 2020). As for hierarchical clustering, according to the direction of clustering, the hierarchical clustering can be divided into agglomerative hierarchical clustering (AHC) and divisive hierarchical K 52,10 clustering (DHC) (Dogan and Birant, 2022); and there have been a considerable amount of literature showing that the AHC algorithms are more widely used than the DHC algorithms since their clustering results are more accurate (Prabhagar and Punniyamoorthy, 2020).…”