Adaptive Monitoring with Dynamic Differential Tracing-Based Diagnosis

Abstract

Abstract. Ensuring high availability, adequate performance, and proper operation of enterprise software systems requires continuous monitoring. Today, most systems operate with minimal monitoring, typically based on service-level objectives (SLOs). Detailed metric-based monitoring is often too costly to use in production, while tracing is prohibitively expensive. Configuring monitoring when problems occur is a manual process.In this paper we propose an alternative: Minimal monitoring with SLOs is used to detect errors. When an error is detected, detailed monitoring is automatically enabled to validate errors using invariant-correlation models. If validated, Application-Response-Measurement (ARM) tracing is dynamically activated on the faulty subsystem and a healthy peer to perform differential trace-data analysis and diagnosis.Based on fault-injection experiments, we show that our system is effective; it correctly detected and validated errors caused by 14 out of 15 injected faults. Differential analysis of the trace data collected for 210 seconds allowed us to top-rank the faulty component in 80% of the cases. In the remaining cases the faulty component was ranked within the top-7 out of 81 components. We also demonstrate that the overhead of our system is low; given a false positive rate of one per hour, the overhead is less than 2.5%.

Diagnosis of recurrent faults using log files

“…In this work we focus on failures that have a clear manifestation in log files rather than failures that do not generate events in log files. In particular, failures that are performance-related can be detected and diagnosed by other means as we have shown in our prior work [21]. We describe in Section 3.6, how to identify fault classes that do not have a clear manifestation in log files and merge those fault classes to improve the classification performance.…”

“…Once a failure is validated (e.g., as described in previous work [12,13,21]), we seek to identify recurrent faults that underlie the observed failure. If the fault that triggered a failure has been seen before, we may be able to fast-track resolution by retrieving information about past actions to restore the failed component.…”

Goal-Oriented Monitoring Adaptation: Methodology and Patterns

“…Collecting additional metrics or joining managed resources is addressed in [2,3,5] either to adapt monitoring to meet SLA modifications, or to deal with the managed scope changes, or even to operate a "minimal" monitoring that is able to be extended in case of SLA violations. Indeed, the capability of scaling up/down the monitored metrics and resources is important as an adaptation action.…”

“…This section enumerates some existing trends focusing on (i) adapting the QoS monitoring in autonomic systems [2,3,4,5,6], and (ii) designing patterns regarding the distributed deployment as well as the adaptation of MAPE loop modules [7,8,9].…”

An extensible framework for repair-driven monitoring

“…The diagnostic approaches differ by monitoring data and overhead. To localize abnormal components from application traces we refer the reader to [16], for identifying recurrent faults from log-files to [13], [14], and finally for localizing faulty components the reader is referred to [15], [17]- [19].…”