Advances and Challenges for Scalable Provenance in Stream Processing Systems

Abstract

Abstract. While data provenance is a well-studied topic in both database and workflow systems, its support within stream processing systems presents a new set of challenges. Part of the challenge is the high stream event rate and the low processing latency requirements imposed by many streaming applications. For example, emerging streaming applications in healthcare or finance call for data provenance, as illustrated in the Century stream processing infrastructure that we are building for supporting online healthcare analytics. At anytime, given an output data element (e.g., a medical alert) generated by Century, the system must be able to retrieve the input and intermediate data elements that led to its generation. In this paper, we describe the requirements behind our initial implementation of Century's provenance subsystem. We then analyze its strengths and limitations and propose a new provenance architecture to address some of these limitations. The paper also includes a discussion on the open challenges in this area.

Analysis of Memory Constrained Live Provenance

“…It can be categorized with coarse-grained provenance methods that identify dependencies between streams or sets of streams [27,26], and fine-grained methods that identify dependencies among individual stream elements [23,10,18,22]. Sansrimahachai et al [23] propose the Stream Ancestor Function -a reverse mapping function to express precise dependencies between input and output stream elements (fine-grained).…”

Comparative Study of Evaluating the Trustworthiness of Data Based on Data Provenance

“…Bertino and his associates [12,13,36] developed and further improved the frameworks that evaluate the trustworthiness of data based on data provenance. Data provenance has also been investigated in online health care analytics for a biomedical data stream system in IBM's Century [37,38]. Malaverri et al [39] introduced a provenance-based approach for evaluating data managed by E-Science applications.…”

Towards Automatic Capturing of Semi-structured Process Provenance

“…Further, the acquired provenance information could be made accessible in different provenance models like e.g. the Open Provenance Model (OPM) [12] or the value centric model (TVC) [13]. However these are just alternative representations of the derived provenance data, while the focus of this paper is on acquiring the provenance data rather then how to represent them.…”