Efficient Online Log Parsing with Log Punctuations Signature

Zhang, Shijie; Wu, Gang

doi:10.3390/app112411974

Cited by 7 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The average PA achievable against the 16 log datasets in LogHub has improved with the introduction of recent parsers such as Paddy [25], SwissLog [28], LTmatch [24], LogPunk [204], Drain+ [196], Hue [199], and Brain [192]. However, the improvement margin has decreased due to the higher overall level of accuracy demonstrated by modern methods in general (Fig.…”

Section: Online Log Parser Performancementioning

confidence: 99%

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Lupton,

Washizaki,

Yoshioka

et al. 2024

IEEE Access

View full text Add to dashboard Cite

As production system estates become larger and more complex, ensuring stability through traditional monitoring approaches becomes more challenging. Rule-based monitoring is common in industrial settings, but it has limitations. These include the difficulty of crafting rules capable of detecting unforeseen issues and the burden of manually maintaining rule sets. A potential solution to effectively manage complex system states is log anomaly detection. Workflows for log anomaly detection utilize several fundamental components. These include preprocessors for data cleansing, parsers to extract structured information from raw log data, encoding algorithms to convert extracted data into usable model input features, anomaly detection methods to isolate anomalous signals, and feedback mechanisms to incrementally improve model performance. This study explores the current state of research into online parser-supported log anomaly detection methods, investigates recent research trends, compares the performances of parser and anomaly detection methods using common public datasets and metrics, and assesses their performance evolution over time. Additionally, it classifies available methods using a newly introduced taxonomy, highlights current research gaps, and recommends future research directions.INDEX TERMS Log parsing, log template extraction, online algorithms, anomaly detection.

show abstract

Section: Online Log Parser Performancementioning

confidence: 99%

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Lupton,

Washizaki,

Yoshioka

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Logan [1] is a distributed online log parser, which is also based on the Longest Common Subsequence algorithm. LogPunk 2021 [34] and QuickLogS [9] are two notable hash-like methods for log parsing. LogStamp [27] is a sequence labeling-based automatic online log parsing method.…”

Section: Pattern Aware Log Parsingmentioning

confidence: 99%

Interactive Log Parsing via Light-weight User Feedbacks

Wang¹,

Xie²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Template mining is one of the foundational tasks to support log analysis, which supports the diagnosis and troubleshooting of large scale Web applications. This paper develops a human-in-the-loop template mining framework to support interactive log analysis, which is highly desirable in real-world diagnosis or troubleshooting of Web applications but yet previous template mining algorithms fails to support it. We formulate three types of light-weight user feedbacks and based on them we design three atomic humanin-the-loop template mining algorithms. We derive mild conditions under which the outputs of our proposed algorithms are provably correct. We also derive upper bounds on the computational complexity and query complexity of each algorithm. We demonstrate the versatility of our proposed algorithms by combining them to improve the template mining accuracy of five representative algorithms over sixteen widely used benchmark datasets.

show abstract

“…In addition, they cannot detect all the templates correctly, which propagates through the learning model and influences the results. To increase the accuracy, LogPunk [53] selects candidates based on punctuation marks; only then are templates extracted. There is also an effort to build a log parser taking semantic features into consideration [54] so that information about parameters is not omitted but is integrated into templates.…”

Section: Dealing With Concept Driftmentioning

confidence: 99%

Software Failure Log Analysis for Engineers—Review

2023

View full text Add to dashboard Cite

The use of automated methods for log analysis is unavoidable in any large company; therefore, it has attracted attention from engineers and researchers. As a result, the number of articles in the field grows yearly and new approaches are frequently proposed. Unfortunately, published research works only sometimes meet the needs of engineers wishing to apply the methods in real-life systems. A common issue is that the method’s benefits often do not compensate for the effort required for its implementation and maintenance. Therefore, engineers must understand the pros and cons of full-scale applications, including the implementation details and the required effort. This work provides a comprehensive review of automated log analysis methods and aims to offer a guide for software engineers who fix integration and production failures. The article categorizes and provides an overview of existing methods and assesses their implementation and maintenance costs, as well as the feasibility of the methods. The article also identifies and describes the shortcomings of existing methods, including concept drift, which is not addressed with sufficient attention, as well as the lack of online benchmarks and the interpretation of the log sequence as a language, without an in-depth analysis of its properties. Despite growing efforts to provide feasible and widely adopted solutions, many reference implementations are unavailable. Consequently, the time and computation complexities differ between various implementations of the same approaches, making the results of research work difficult to replicate in real-life scenarios.

show abstract

Efficient Online Log Parsing with Log Punctuations Signature

Cited by 7 publications

References 30 publications

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Interactive Log Parsing via Light-weight User Feedbacks

Software Failure Log Analysis for Engineers—Review

Contact Info

Product

Resources

About