Transactional stream processing

Abstract

Many stream processing applications require access to a multitude of streaming as well as stored data sources. Yet there is no clear semantics for correct continuous query execution over these data sources in the face of concurrent access and failures. Instead, today's Stream Processing Systems (SPSs) hard-code transactional concepts in their execution models, making them both hard to understand and inflexible to use. In this paper, we show that we can successfully reuse the traditional transactional theory (with some minimal extensions) in order to cleanly define the correct interaction of a set of continuous and one-time queries concurrently accessing both streaming and stored data sources. The result is a unified transactional model (UTM) for query processing over streams as well as traditional databases. We present a transaction manager that implements this model on top of an existing storage manager for streams (MXQuery/SMS). Experiments on the Linear Road Benchmark show that our transaction manager flexibly ensures correctness in case of concurrency and failures, without sacrificing from performance. Moreover, this model is powerful enough to express the implicit transactional behaviors of a representative set of state-of-the-art SPSs.

DAGGTAX: A Taxonomy of Data Aggregation Processes

“…A certain form of concurrency control needs to be implemented to achieve a desired level of consistency. For example, the aggregate process may achieve full isolation from other processes, i.e., they can only see the aggregated result when the DAP completes, using serializable concurrency control [36]. To improve performance or timeliness, one may choose a less stringent concurrency control that allows other processes to access the sub-aggregate results of the DAP, which may lead to a less accurate final result.…”

Data stream processing with concurrency control

“…A general transaction model for data streams is studied by Botan et al [5]. They treated both streaming data and non-streaming data equally as objects to which read and write operations are defined.…”

“…However, such resources tend to be shared and updated by other systems (Figure 1), and conventional DSMSs could produce improper results in such situation. Surprisingly, in the field of data stream processing, most of the research about the use of external resources do not allow systems to update resources [7,[21][22][23], and so far only a few attempts have been made for such situations [5,9]. In this paper, we consider situations where systems are allowed to share and update resources, and address the problem inherent in these situations.…”

Fault Tolerant Distributed Stream Processing based on Backtracking

“…Therefore, instead of instant consistency of global state, the goal of transactional stream processing is to keep the eventual consistency [3]; this difference raises specific technical challenges. In the context of database transactions, the previous work on transactional stream processing was mostly based on the notion of snapshot isolation [2,4,6,7,10] with the motivation to split a stream into a sequence of bounded chunks in order to apply the semantics of database transaction to each chunk, i.e. put the operation on a data chunk in a transaction boundary to yield a state snapshot.…”