Log-Based Anomaly Detection with the Improved K-Nearest Neighbor

Abstract

Logs play an important role in the maintenance of large-scale systems. The number of logs which indicate normal (normal logs) differs greatly from the number of logs that indicate anomalies (abnormal logs), and the two types of logs have certain differences. To automatically obtain faults by K-Nearest Neighbor (KNN) algorithm, an outlier detection method with high accuracy, is an effective way to detect anomalies from logs. However, logs have the characteristics of large scale and very uneven samples, which will affect the results of KNN algorithm on log-based anomaly detection. Thus, we propose an improved KNN algorithm-based method which uses the existing mean-shift clustering algorithm to efficiently select the training set from massive logs. Then we assign different weights to samples with different distances, which reduces the negative effect of unbalanced distribution of the log samples on the accuracy of KNN algorithm. By comparing experiments on log sets from five supercomputers, the results show that the method we proposed can be effectively applied to log-based anomaly detection, and the accuracy, recall rate and F measure with our method are higher than those of traditional keyword search method.

Two Class Pruned Log Message Anomaly Detection

“…The results with Auto-LSTM, IKNN and nLSALog for the BGL data set are given in Table 2 a. The precision, recall and F-measure results for negative logs are better than the 92%, 91% and 92%, respectively, with the improved K-nearest neighbors (IKNN) supervised algorithm [ 36 ]. The precision, recall and F-measure results for negative logs are also better than the 82.5%, 94.7% and 88.2%, respectively, with the nLSALog algorithm [ 40 ].…”

“…A decision tree model was considered in [32] to detect faults using log messages. An improved supervised K-nearest neighbors (IKNN) method was employed in [36] to detect anomalies in log messages. However, using supervised methods is not always possible because of the lack of labeled data.…”

Log Message Anomaly Detection with Oversampling

“…The F-measure is 99.6% for both negative and positive logs. The precision, recall, and F-measure with oversampling for negative logs are better than the 96%, 96%, and 96%, respectively, with the Improved K-Nearest Neighbor algorithm [23].…”

Big Data Visualization for Cyber Security: BETH Dataset

“…Liberty veri setinin K-en yakın komşular makine öğrenimi algoritması ile log tabanlı anomali tespiti için 2020 yılına ait bir çalışma kullanıldığı görülmektedir [15]. 2021 yılında gerçekleştirilen başka bir araştırmada log verisindeki anomalilerin taksonomisi konusu araştırılmış ve Thunderbird, Spirit ve BGL veri setlerinin kullanıldığı görülmüştür [16]. Blue Gene/L, Thunderbird, Redstorm, Liberty ve Spirit veri setlerinin oluşturulmaya başlanma tarihleri oldukça eskidir [17].…”