Log Anomaly Detection Based on Hierarchical Graph Neural Network and Label Contrastive Coding

Abstract

System logs are essential for detecting anomalies, querying faults, and tracing attacks. Because of the time-consuming and labor-intensive nature of manual system troubleshooting and anomaly detection, it cannot meet the actual needs. The implementation of automated log anomaly detection is a topic that demands urgent research. However, the prior work on processing log data is mainly one-dimensional and cannot profoundly learn the complex associations in log data. Meanwhile, there is a lack of attention to the utilization of log labels and usually relies on a large number of labels for detection. This paper proposes a novel and practical detection model named LCC-HGLog, the core of which is the conversion of log anomaly detection into a graph classification problem. Semantic temporal graphs (STG) are constructed by extracting the raw logs' execution sequences and template semantics. Then a unique graph classifier is used to better comprehend each STG's semantic, sequential, and structural features. The classification model is trained jointly by graph classification loss and label contrastive loss. While achieving discriminability at the class-level, it increases the fine-grained identification at the instance-level, thus achieving detection performance even with a small amount of labeled data. We have conducted numerous experiments on real log datasets, showing that the proposed model outperforms the baseline methods and obtains the best all-around performance. Moreover, the detection performance degrades to less than 1% when only 10% of the labeled data is used. With 200 labeled samples, we can achieve the same or better detection results than the baseline methods.

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

“…4). LCC-HGLog has achieved the highest recorded F-score (99.9%) [72]. Many other methods have realized F-scores above 99%, starting with LogRobust in late 2019 [227].…”

Semantic Hierarchical Classification Applied to Anomaly Detection Using System Logs with a BERT Model

“…Moreover, the use of hierarchical classification models can improve the accuracy of anomaly detection by considering the latent structure of semantic relationships in the data [15,16]. These approaches allow for a more effective capture of contextual and thematic information in system logs, which can be crucial for accurately identifying anomalous behavior [17,18].…”