LogPal: A Generic Anomaly Detection Scheme of Heterogeneous Logs for Network Systems

Abstract

As a key resource for diagnosing and identifying problems, network syslog contains vast quantities of information. And it is the main source of data for anomaly detection of systems. Syslog presents the characteristics of large scale, diverse types and sources, data noise, and quick evolvement, which makes the detection methods not generic enough. To effectively address problem of log anomaly labelling caused by massive heterogeneous logs, we propose LogPal, a generic anomaly detection scheme of heterogeneous logs for network systems, which innovatively combines template sequences and raw log sequences to construct and generate log pattern events. By improving the self-attention mechanism of transformer, LogPal proactively synthesizes self-attention and handles log pattern events in a unique way. The model can make full use of log template and sequence semantic information, by automatically becoming aware of the pattern of logs. We implemented experiments to evaluate the performance of LogPal on publicly available datasets, and the outcome of the experiments shows that LogPal automatically adapts to log type changes and improves precision, recall, and F1 score to 99% on publicly available datasets.

An Intelligent Framework for Log Anomaly Detection Based on Log Template Extraction

“…The former entails calculating the similarity between log strings using measures such as Euclidean distance, Cosine similarity, or Levenshtein distance. In this study, we employ the latter approach, specifically the log template extraction method based on FT-Tree, which has been demonstrated to be accurate and efficient to extract templates from logs (Niu et al, 2023;Li & Su, 2023;Sun & Xu, 2023). The rationale behind selecting the FT-Tree model lies in its ability to maintain high accuracy while incurring low computational costs.…”