Paddy: An Event Log Parsing Approach using Dynamic Dictionary

Huang, Shaohan; Liu, Yi; Fung, Carol; He, Rong; Zhao, Yining; Yang, Hailong; Luan, Zhongzhi

doi:10.1109/noms47738.2020.9110435

Cited by 26 publications

(26 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Paddy system [9] is the first to employ Information Retrieval techniques to address the system log parsing problem. It uses a dynamic inverted index data structure to store all the tokens present in the log message text in the form of terms.…”

Section: Information Retrieval Techniquesmentioning

confidence: 99%

“…Then the possible candidate event IDs are retrieved and the similarity between the input log message and each candidate template is calculated using a custom similarity metric called fitting score. Fitting score is the weighted sum of Jaccard similarity and length feature [9]. If the highest similarity among the similarity values of all of the candidates is not greater than an experimentally decided threshold, the input log message is considered as a new template and gets indexed into the dynamic dictionary.…”

Section: Information Retrieval Techniquesmentioning

confidence: 99%

“…When extracting a representative template for the log messages generated by the same event, the positions of variable parts in the event template are replaced by a wildcard. Related literature has reported "*" [4], [8] and "< * >" [1], [9] as the most used wildcards. In this paper, "< * >" is used to represent the wildcard.…”

Section: Introductionmentioning

confidence: 99%

“…al. [9] used Information Retrieval techniques for log parsing. Using publicly available benchmarks, they experimentally showed that it is the best solution for log parsing with respect to the parsing accuracy.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

vue4logs -- Automatic Structuring of Heterogeneous Computer System Logs

Boyagane¹,

Katulanda²,

Ranathunga³

et al. 2022

Preprint

View full text Add to dashboard Cite

Computer system log data is commonly used in system monitoring, performance characteristic investigation, workflow modeling and anomaly detection. Log data is inherently unstructured or semi-structured, which makes it harder to understand the event flow or other important information of a system by reading raw logs. The process of structuring log files first identifies the log message groups based on the system events that triggered them, and extracts an event template to represent the log messages of each event. This paper introduces a novel method to extract event templates from raw system log files, by using the vector space model commonly used in the field of Information Retrieval to vectorize log data and group log messages into event templates based on their vector similarity. Template extraction process is further enhanced with the use of character and length based filters. When evaluated on publicly available real-world log data benchmarks, this proposed method outperforms all the available state-of-the-art systems in terms of accuracy and robustness. NOTE:

show abstract

Section: Information Retrieval Techniquesmentioning

confidence: 99%

Section: Information Retrieval Techniquesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

vue4logs -- Automatic Structuring of Heterogeneous Computer System Logs

Boyagane¹,

Katulanda²,

Ranathunga³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, we search the signature group one by one, which will seriously affect the efficiency in such cases. Fortunately, inverted index [30] and prefix tree [26] can improve search efficiency.…”

Section: Efficiencymentioning

confidence: 99%

Efficient Online Log Parsing with Log Punctuations Signature

Zhang

2021

Applied Sciences

View full text Add to dashboard Cite

Logs, recording the system runtime information, are frequently used to ensure software system reliability. As the first and foremost step of typical log analysis, many data-driven methods have been proposed for automated log parsing. Most existing log parsers work offline, requiring a time-consuming training progress and retraining as the system upgrades. Meanwhile, the state of the art online log parsers are tree-based, which still have defects in robustness and efficiency. To overcome such limitations, we abandon the tree structure and propose a hash-like method. In this paper, we propose LogPunk, an efficient online log parsing method. The core of LogPunk is a novel log signature method based on log punctuations and length features. According to the signature, we can quickly find a small set of candidate templates. Further, the most suitable template is returned by traversing the candidate set with our log similarity function. We evaluated LogPunk on 16 public datasets from the LogHub comparing with five other log parsers. LogPunk achieves the best parsing accuracy of 91.9%. Evaluation results also demonstrate its superiority in terms of robustness and efficiency.

show abstract