Automated Online Monitoring of Distributed Applications through External Monitors

Abstract

It is a challenge to provide detection facilities for large-scale distributed systems running legacy code on hosts that may not allow fault tolerant functions to execute on them. It is tempting to structure the detection in an observer system that is kept separate from the observed system of protocol entities, with the former only having access to the latter's external message exchanges. In this paper, we propose an autonomous self-checking Monitor system, which is used to provide fast detection to underlying network protocols. The Monitor architecture is application neutral and, therefore, lends itself to deployment for different protocols, with the rulebase against which the observed interactions are matched, making it specific to a protocol. To make the detection infrastructure scalable and dependable, we extend it to a hierarchical Monitor structure. The Monitor structure is made dynamic and reconfigurable by designing different interactions to cope with failures, load changes, or mobility. The latency of the Monitor system is evaluated under fault free conditions, while its coverage is evaluated under simulated error injections.

How to Keep Your Head above Water While Detecting Errors

“…Stateful detection is looked upon as a powerful mechanism for building dependable distributed systems [1] [10]. However, scaling a stateful detection system with increasing rate of messages is a challenge.…”

“…In prior work, we have presented Monitor [10] which provides stateful detection by observing the messages exchanged between application components. Monitor has a breaking point in terms of the rate of messages it has to process.…”

“…For example, the stateful network intrusion detection system (NIDS) Snort running on a general-purpose CPU can process traffic up to 500 Mbps [15]. For Monitor, we have observed that the breaking point on a standard Linux box is around 100 packets/sec [10].…”

“…In random sampling, messages are sampled randomly without looking at the type or content of the message. As shown in [10], under non-sampling conditions, Monitor's accuracy and precision suffer when the rate of incoming messages goes above a particular point which is denoted as R th . Therefore, random sampling is activated at any rate R > R th , in which Monitor drops messages uniformly.…”

“…In previous work we developed Monitor, a framework for online error detection in distributed applications [10]. Online implies the detection happens when the application is executing.…”