Practical whole-system provenance capture

Abstract

Data provenance describes how data came to be in its present form. It includes data sources and the transformations that have been applied to them. Data provenance has many uses, from forensics and security to aiding the reproducibility of scientific experiments. We present CamFlow, a whole-system provenance capture mechanism that integrates easily into a PaaS offering. While there have been several prior whole-system provenance systems that captured a comprehensive, systemic and ubiquitous record of a system's behavior, none have been widely adopted. They either A) impose too much overhead, B) are designed for long-outdated kernel releases and are hard to port to current systems, C) generate too much data, or D) are designed for a single system. CamFlow addresses these shortcoming by: 1) leveraging the latest kernel design advances to achieve efficiency; 2) using a self-contained, easily maintainable implementation relying on a Linux Security Module, NetFilter, and other existing kernel facilities; 3) providing a mechanism to tailor the captured provenance data to the needs of the application; and 4) making it easy to integrate provenance across distributed systems. The provenance we capture is streamed and consumed by tenant-built auditor applications. We illustrate the usability of our implementation by describing three such applications: demonstrating compliance with data regulations; performing fault/intrusion detection; and implementing data loss prevention. We also show how CamFlow can be leveraged to capture meaningful provenance without modifying existing applications.

Unicorn: Runtime Provenance-Based Detector for Advanced Persistent Threats

“…Hosts that exhibit more diverse behaviors, such as workstations [18], pose a greater challenge for IDS in general. Modern provenance capture systems (e.g., CamFlow) help mitigate this issue as they can separate provenance data based on, e.g., namespaces and control groups [100]. However, we acknowledge that endpoint security for workstations presents extra challenges that UNICORN does not attempt to address in this work, as it was originally designed to protect more stable environments.…”

“…Provenance systems construct a single, wholesystem provenance DAG with a partial-order guarantee, which allows for efficient streaming computation ( § IV-B) and fully contextualized analysis ( L2 ). We present UNICORN using CamFlow [100], although it can obtain provenance from other systems, such as LPM [16] and Spade [44], the latter of which interoperates with commodity audit systems such as Linux Audit and Windows ETW. 2 Builds at runtime an in-memory histogram.…”

“…There exist many graph similarity measures, but many approaches (e.g., graph isomorphism) are too restrictive (i.e., require two graphs to be exactly or largely identical) [110], because even normal executions often produce slightly different provenance graphs. Wholesystem provenance graphs can grow large quickly [100], so NP-complete [42], [95] and even polynomial algorithms [71], [122] are too computationally expensive for streaming settings. As we show in the following sections, UNICORN's graph similarity algorithm does not exhibit these problems.…”

“…In the previous section, we show how UNICORN's parameters influence detection performance; this section and the one that follows examine UNICORN's runtime overhead. These sections consider only the CamFlow implementation, which has been shown to produce negligible overhead [100], allowing us to isolate UNICORN's performance characteristics. We use Amazon EC2 i3.2xlarge Linux machines with 8 vCPUs and 61GiB of memory.…”

Runtime Analysis of Whole-System Provenance

“…We built CamQuery on top of the CamFlow provenance capture system [3,79,80], our actively-maintained provenance monitor built as a stackable Linux Security Module (LSM) [69]. Compared to other existing capture techniques [34,72], an LSM-based approach ensures that CamFlow can observe and mediate all information flows between processes and kernel objects [27,31,36,51] (see § 4.2 for further discussion).…”