Exploiting constraints to build a flexible and extensible data stream processing middleware

Abstract

A wide range of real-time applications process stream-based data. To process this stream-based data in an application-independent manner, many stream processing systems have been built. However, none of them reached a huge domain of applications, such as databases did. This is due to the fact that they do not consider the specific needs of real-time applications. For instance, an application which visualizes streambased data has stringent timing constraints, or may even need a specific hardware environment to smoothly process the data. Furthermore, users may even add additional constraints. E.g., for security reasons they may want to restrict the set of nodes that participates in processing. Thus, constraints naturally arise on different levels of query processing.In this work we classify constraints that occur on different levels of query processing. Furthermore we propose a scheme to classify the constraints and show how these can be integrated into the query processing of the distributed data stream middleware NexusDS.

Tool support for the design and management of context models

“…To find suitable processing nodes, operators must be annotated by technical context describing its requirements in terms of hard-and software resources [31]. This meta data is also managed by the repository and is used during the transformation to the execution model for constraining the selection of suitable nodes for the specific operator.…”

“…Operators in the domain of NexusDS have special needs that must be taken into account when modeling the operator's meta data definition [31]. From a data modeling point of view the following requirements must be satisfied:…”

Sicherer Zugriff und sichere Verarbeitung von Kontextdatenströmen in einer verteilten Umgebung

“…Das Datenstromverarbeitungssystem, welches die Datenintegration und Datenverarbeitung vornimmt, muss also eine Zugriffskontrolle ermöglichen [4]. Da die denkbaren Anwendungsszenarien, wie in [8] beschrieben, mannigfaltig sind, muss das System offen in Bezug auf zukünftige Anwendungen sowie der zu verarbeitenden Daten sein. Das bedeutet insbesondere, dass Daten sowie notwendige dedizierte Verarbeitungstechniken, in Form von Operatoren, integriert werden sollen, um Anwendungen geeignet unterstützen zu können.…”

M-Top

“…As argued in [3], an adaptation problem in data stream processing systems persists which makes it difficult for applications (relying on the same processing paradigm-i.e., push-based stream processing) to seamlessly integrate custom functionality and to affect the distribution process accordingly. Tight application integration in data stream processing systems is beneficial because data processing is tightened together and the actual processing happens near to the actual data.…”

“…Consider the example of an interactive visualization application as sketched in [3]. The application visualizes the result of a continuous query on data streams exploiting the functionality of distributed stream processing systems.…”