Log20

Abstract

When systems fail in production environments, log data is often the only information available to programmers for postmortem debugging. Consequently, programmers' decision on where to place a log printing statement is of crucial importance, as it directly affects how effective and efficient postmortem debugging can be. This paper presents Log20, a tool that determines a near optimal placement of log printing statements under the constraint of adding less than a specified amount of performance overhead. Log20 does this in an automated way without any human involvement. Guided by information theory, the core of our algorithm measures how effective each log printing statement is in disambiguating code paths. To do so, it uses the frequencies of different execution paths that are collected from a production environment by a low-overhead tracing library. We evaluated Log20 on HDFS, HBase, Cassandra, and ZooKeeper, and observed that Log20 is substantially more efficient in code path disambiguation compared to the developers' manually placed log printing statements. Log20 can also output a curve showing the trade-off between the informativeness of the logs and the performance slowdown, so that a developer can choose the right balance. CCS CONCEPTS • Computer systems organization → Reliability; • Software and its engineering → System administration;

Quality Properties of Execution Tracing, an Empirical Study

“…The majority of the publications with regard to the problem they address can be classified in three groups: (1) where to insert log statements in the source code [13,20,[23][24][25][26][27][28], (2) what to log to provide sufficient information for failure analysis [13,17,27,29], and (3) how to log [13,19,27]. Tool support for selecting the appropriate places in the source code for inserting log statements consistently or for achieving optimal performance by the trade-off in the amount of information logged and the speed of execution usually involve more than one group mentioned above [13,17,[25][26][27][28][29].…”

“…Yuan et al and Zhao et al confirm that log messages are frequently the only tools to analyse system failures in the field in a postmortem way [13,27]. Yuan et al study the efficiency of logging practices in five large software systems, with randomly selected 250 failure reports, and the corresponding source code; moreover, they report that postmortem analysis caused problems in 57% of the cases based on the provided log data [13].…”

“…whether the performance of the trace is appropriate with regard to the storage, (10) whether the structure of the trace output is the same at the different components and applications in the same application domain, (11) whether the trace mechanism offers automation feature such as filtering duplicate entries, (12) whether standard trace implementations are used (e.g., log4j), (13) whether the trace mechanism can deal with increasing data volume, (14) whether the trace mechanism is capable of dealing with parallel execution, (15) whether the trace mechanism is capable of measuring performance (a) at different levels of the architecture (b) with different granularities and (c) not only the response time but also the quality of data returned can be investigated based on the trace, (16) whether the trace mechanism offers the possibility to make queries on timestamps, (17) whether the trace mechanism is capable of making a prediction about the system's availability based on the traces, (18) whether the trace mechanism is capable of generating usage statistics, (19) whether the trace mechanism is capable of measuring quality of service metrics for SLAs, (20) whether the trace code insertion can be done automatically in the application, (21) whether the trace mechanism is capable of replaying and simulating sequence of actions based on the traces, (22) whether the trace data are available as the events occur not only after the trace session is finished, (23) whether the trace data are available also through APIs, (24) whether the output format of the trace can be changed including DB and text file formats, (25) whether the trace code can be located in separate modules and not intermingled into the application code, (26) whether the trace mechanism is capable of correlating actions on different nodes in the case of distributed application, (27) whether the tracing mechanism is capable of keeping its perturbations on a possible minimum level with regard to the application, (28) whether the CPU usage can be traced, (29) the memory usage can be traced, (30) whether the trace mechanism is capable of tracing the following properties in the case of real-time applications: (a) task switches, which task, when, (b) interrupts, (c) tick rate, (d) CPU usage, (e) memory usage, (f) network utilisation, (g) states of the real-time kernel, which tasks are waiting to execute (waiting queue), which tasks are running, which tasks are blocked. Definition: The variable defines how secure the execution tracing mechanism and its outputs are, i.e., it shows to what extent the execution tracing mechanism and its outputs are exposed to vulnerabilities and how likely it is that sensitive information leaks out through them.…”

“…Log the branches of unusual input or environment effects that might not be covered by testing Zhao et al implemented tool support for placing log statements in the code without human intervention and without the need of domain knowledge [27]. Their tool collects execution path information with a low-overhead logging library and does an evaluation about the frequencies of different execution paths to compute the optimal places for log statements for the developer to be able to distinguish between those paths [27].…”

A Theoretical Framework for Understanding the Relationship Between Log Parsing and Anomaly Detection

“…The definition only assumes that normal and abnormal behaviors of the system are known and distinguishable in logs. This assumption can be easily satisfied when the accuracy of anomaly detection techniques, including log parsing techniques, are evaluated in controlled experiments using known benchmarks; furthermore, enhancing the quality of logs to distinguish normal and abnormal behaviors has been actively studied [11,[22][23][24]. In the rest of the paper we adopt this definition and make its underlying assumption.…”