Distributed time-aware provenance

Abstract

The ability to reason about changes in a distributed system's state enables network administrators to better diagnose protocol misconfigurations, detect intrusions, and pinpoint performance bottlenecks. We propose a novel provenance model called Distributed Time-aware Provenance (DTaP) that aids forensics and debugging in distributed systems by explicitly representing time, distributed state, and state changes. Using a distributed Datalog abstraction for modeling distributed protocols, we prove that the DTaP model provides a sound and complete representation that correctly captures dependencies among events in a distributed system. We additionally introduce DistTape, an implementation of the DTaP model that uses novel distributed storage structures, query processing, and cost-based optimization techniques to efficiently query time-aware provenance in a distributed setting. Using two example systems (declarative network routing and Hadoop MapReduce), we demonstrate that DistTape can efficiently maintain and query time-aware provenance at low communication and computation cost.

Explaining outputs in modern data analytics

“…An intuitive way to compute provenance for output tuples is the approach followed by [29,46], where each rule is rewritten as first determining a binding of all variables involved, and then yielding the contribution to the output relation. For example, the second rule above can be rewritten as:…”

“…Data provenance is a versatile concept and has also been used for network management, most notably in ExS-PAN [47] and DTaP [46]. Both these systems are based on NDlog, a variation of Datalog with aggregation (e.g., MIN), which was first introduced in [32] for the declarative definition and analysis of routing protocols.…”

“…Over the last decade, the idea of provenance received considerable attention in a wide range of application areas, including data curation [8], system debugging [46,3,29], access control [17], workflow analysis [20,4,42,26], probabilistic data management [6], network management [47,43], data integration [12,28], and query result explanation [37], among others. At the same time, the scale and scope of modern data analytics have moved past what traditional databases were designed to handle.…”

“…However, we know that there is a succinct explanation for why A and B are connected; specifically, the edge (A,B). Existing systems [29,46] with backward tracing support for the previous computation return the whole set of paths between A and B; this is prohibitive for large connected components and, more importantly, the explanation is very difficult for a human to digest.…”

“…Suppressing records ensures that each operator is correctly re-simulated, but does not solve the issue that the computation's input does not yield the desired output, which hinders the logical debugging of the program. In fact, all provenance-aware systems [26,27,4,31,46,29,1] provide explanations that guarantee the reproduction of the output only when the conditions of Theorem 1 are satisfied.…”