Live Debugging of Distributed Systems

Abstract

Abstract. Debugging distributed systems is challenging. Although incremental debugging during development finds some bugs, developers are rarely able to fully test their systems under realistic operating conditions prior to deployment. While deploying a system exposes it to realistic conditions, debugging requires the developer to: (i) detect a bug, (ii) gather the system state necessary for diagnosis, and (iii) sift through the gathered state to determine a root cause. In this paper, we present MaceODB, a tool to assist programmers with debugging deployed distributed systems. Programmers define a set of runtime properties for their system, which MaceODB checks for violations during execution. Once MaceODB detects a violation, it provides the programmer with the information to determine its root cause. We have been able to diagnose several non-trivial bugs in existing mature distributed systems using MaceODB; we discuss two of these bugs in this paper. Benchmarks indicate that the approach has low overhead and is suitable for in situ debugging of deployed systems.

Predicting and preventing inconsistencies in deployed distributed systems

“…Their approach involves launching queries across the distributed system that is described and deployed using the OverLog/P2 [Singh et al 2006] declarative language/runtime combination. D 3 S [Liu et al 2008] and MaceODB [Dao et al 2009] enable developers to specify global predicates which are then automatically checked in a deployed distributed system. By using binary instrumentation, D 3 S can work with legacy systems.…”

Inferring models of concurrent systems from logs of their behavior with CSight

“…Runtime monitoring approaches can detect execution sequence or data property violations for error reporting [50,27,41,20] or selfadaptation [40,23,58,47]. CSight models can be used at runtime to verify that executions conform to the expected model.…”

Distributed debugging for mobile networks

“…As such, programmers have to run over and over a program until the bug would appear, making the reproduction of a rare erroneous condition even rarer. Moreover, as remarked by Dao et al in [45], reproducing the spectrum of possible states that a distributed application can be in and exposing the application to them before its deployment may be not feasible. This is more exacerbated in ambientoriented application due to the dynamic environment in which they run.…”