Modeling the execution semantics of stream processing engines with SECRET

Dindar, Nihal; Tatbul, Nesime; Miller, Renée J.; Haas, Laura M.; Botan, Irina

doi:10.1007/s00778-012-0297-3

Cited by 33 publications

(24 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For ease of explanation, we present the SECRET formalism with reference to time windows, and we briefly discuss the differences in the case of count windows at the end of the section. The interested reader can refer to complete formalization of SECRET for further details [4].…”

Section: Secretmentioning

confidence: 99%

“…The declarative language of Azure Stream analytics that we analyzed in this paper also derives from this processing model. The SECRET model we adopt in this paper was originally designed to capture the processing semantics of this kind of systems [2,4].…”

Section: Processing Streams Of Datamentioning

confidence: 99%

“…The paper contributes to the research on SPEs in several ways: (1) it provides a precise modeling of five modern SPEs using SECRET; (2) it compares the systems and highlights their similarities and differences, thus helping the users to identify the systems that better satisfy their requirements; (3) it identifies some aspects of modern SPEs that SECRET cannot fully capture; (4) it discusses the evolution of SPEs since the definition of SECRET and suggests promising directions for future modeling efforts.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Defining the execution semantics of stream processing engines

et al. 2017

View full text Add to dashboard Cite

IntroductionSeveral modern data-intensive applications need to process large volumes of data on the fly as they are produced. Examples range from credit card fraud detection systems, which analyze massive streams of credit card transactions to identify suspicious patterns, to environmental monitoring applications that continuously analyze sensor data, to click stream analysis of Web sites that identify frequent patterns of interactions. More in general, stream processing is a central requirement in today's information systems.This state of facts pushed the development of several stream processing engines (SPEs) that continuously analyze streams of data to produce new results as new elements enter the streams. Unfortunately, existing SPEs adopt different processing models and standardized execution semantics have not yet emerged. This severely hampers the usability AbstractThe ability to process large volumes of data on the fly, as soon as they become available, is a fundamental requirement in today's information systems. Modern distributed stream processing engines (SPEs) address this requirement and provide low-latency and high-throughput data stream processing in cluster platforms, offering high-level programming interfaces that abstract from low-level details such as data distribution and hardware failures. The last decade saw a rapid increase in the number of available SPEs. However, each SPE defines its own processing model and standardized execution semantics have not emerged yet. This paper tackles this problem and analyzes the execution semantics of some widely adopted modern SPEs, namely Flink, Storm, Spark Streaming, Google Dataflow, and Azure Stream Analytics. We specifically target the notions of windowing and time, traditionally considered the key distinguishing factors that characterize the behavior of SPEs. We rely on the SECRET model, introduced in 2010 to analyze the windowing semantics for the SPEs available at that time. We show that SECRET models well some aspects of the behavior of modern SPEs, and we shed light on the evolution of SPEs after the introduction of SECRET by analyzing the elements that SECRET cannot fully capture. In this way, the paper contributes to the research in the area of stream processing by: (1) contrasting and comparing some widely used modern SPEs based on a formal model of their execution semantics; (2) discussing the evolution of SPEs since the introduction of the SECRET model; (3) suggesting promising research directions to direct further modeling efforts. Affetti et al. J Big Data (2017) et al. J Big Data (2017) 4:12 and interoperability of SPEs, since a user needs to understand system-specific aspects to confront various alternatives and select the ones that better suite her needs. SURVEY PAPERPage 2 of 24 AffettiThe main factors that differentiate the behaviors of SPEs are the models of windows and time they adopt [1]. Windows enable computations that would be otherwise unfeasible on unbounded datasets such as streams. For instance, counting the number o...

show abstract

Section: Secretmentioning

confidence: 99%

Section: Processing Streams Of Datamentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Defining the execution semantics of stream processing engines

et al. 2017

View full text Add to dashboard Cite

show abstract

“…SECRET is (arguably loosely) named after the four dimensions it considers; scope, content, report and tick. Dindar et al (2013) consider these four dimensions with SECRET in their coverage of the heterogeneity of the Coral8, STREAM, StreamBase, and Oracle CEP stream processing engines.…”

Section: Secretmentioning

confidence: 99%

“…Lastly, tick deals with the control loop of a stream processing engine, and in particular when it acts on a given input stream. Given these four dimensions, Dindar et al (2013) consider both time-based and tuple-based windows for the aforementioned stream processing engines.…”

Section: Secretmentioning

confidence: 99%

Spatio-Temporal Stream Reasoning with Adaptive State Stream Generation

Leng¹

2017

View full text Add to dashboard Cite

A lot of today's data is generated incrementally over time by a large variety of producers. This data ranges from quantitative sensor observations produced by robot systems to complex unstructured human-generated texts on social media. With data being so abundant, making sense of these streams of data through reasoning is challenging. Reasoning over streams is particularly relevant for autonomous robotic systems that operate in a physical environment. They commonly observe this environment through incremental observations, gradually refining information about their surroundings. This makes robust management of streaming data and its refinement an important problem. Many contemporary approaches to stream reasoning focus on the issue of querying data streams in order to generate higher-level information by relying on wellknown database approaches. Other approaches apply logic-based reasoning techniques, which rarely consider the provenance of their symbolic interpretations. In this thesis, we integrate techniques for logic-based spatio-temporal stream reasoning with the adaptive generation of the state streams needed to do the reasoning over. This combination deals with both the challenge of reasoning over streaming data and the problem of robustly managing streaming data and its refinement. The main contributions of this thesis are (1) a logic-based spatio-temporal reasoning technique that combines temporal reasoning with qualitative spatial reasoning; (2) an adaptive reconfiguration procedure for generating and maintaining a data stream required to perform spatio-temporal stream reasoning over; and (3) integration of these two techniques into a stream reasoning framework. The proposed spatio-temporal stream reasoning technique is able to reason with intertemporal spatial relations by leveraging landmarks. Adaptive state stream generation allows the framework to adapt in situations in which the set of available streaming resources changes. Management of streaming resources is formalised in the DyKnow model, which introduces a configuration life-cycle to adaptively generate state streams. The DyKnow-ROS stream reasoning framework is a concrete realisation of this model that extends the Robot Operating System (ROS). DyKnow-ROS has been deployed on the SoftBank Robotics NAO platform to demonstrate the system's capabilities in the context of a case study on run-time adaptive reconfiguration. The results show that the proposed system-by combining reasoning over and reasoning about streams-can robustly perform spatio-temporal stream reasoning, even when the availability of streaming resources changes.

show abstract