Reactive stream processing for data-centric publish/subscribe

Abstract

The Internet of Things (IoT) paradigm has given rise to a new class of applications wherein complex data analytics must be performed in real-time on large volumes of fastmoving and heterogeneous sensor-generated data. Such data streams are often unbounded and must be processed in a distributed and parallel manner to ensure timely processing and delivery to interested subscribers. Dataflow architectures based on event-based design have served well in such applications because events support asynchrony, loose coupling, and helps build resilient, responsive and scalable applications. However, a unified programming model for event processing and distribution that can naturally compose the processing stages in a dataflow while exploiting the inherent parallelism available in the environment and computation is still lacking. To that end, we investigate the benefits of blending Reactive Programming with data distribution frameworks for building distributed, reactive, and high-performance stream-processing applications. Specifically, we present insights from our study integrating and evaluating Microsoft .NET Reactive Extensions (Rx) with OMG Data Distribution Service (DDS), which is a standardsbased publish/subscribe middleware suitable for demanding industrial IoT applications. Several key insights from both qualitative and quantitative evaluation of our approach are presented.

Big Data Semantics

“…Figure 1 offers a synthetic view on the pipeline adopted as a reference for this paper. Markl [14] x x Ardagna et al [34] x x x x x x Azzini et al [23] x x Smith et al [26] x x Liao et al [35] x x Duggan et al [36] x x Sowmya et al [37] x x Zhou et al [38] x x Akoush et al [39] x x Glavic [40] x x Berti-Equille et al [41] x x Kläs et al [42] x x Daiber et al [43] x x Shin et al [44] x x x Chiticariu et al [45] x x Fuhring et al [46] x x Bondiombouy et al [47] x x Bergamaschi et al [48] x x Ramakrishnan et al [49] x x Masseroli et al [50] x x Scannapieco et al [51] x x Gualtieri et al [52] x x Liu et al [53] x x Gulzar et al [54] x x De Wit [55] x x Zardetto et al [56] x x Gonzalez et al [57] x x Junghanns et al [58] x x Yu et al [59] x x You et al [60] x x Hagedorn et al [61] x x Kornacker et al [62] x x Costea et al [63] x x Schätzle et al [64] x x Cudré-Mauroux et al [65] x x Appice et al [66] x x Khare2015 et al [67] x x Poggi et al [68] x x Um et al [69] x x Poole et al [16] x x Smith et al [26] x x Gies...…”

“…For this purpose a typical publish/subscribe pattern is used. Khare et al show the application of such an approach in the IoT domain in [67]. In [68] semantic models are used to provide a unified view of the heterogeneous elements composing these systems, and reasoning mechanisms are leveraged to drive adaptation strategies.…”

Exploring the role of sequential computation in distributed systems: motivating a programming paradigm shift

“…In this case, action simply returns last (located on the server); this value will be sent and used as an argument to notify on the client. Our compiler, when given the two input scenarios Send and Notify, produces the same code as the Actor implementation given in the previous section ( § 3.2.4), including the new class for notification messages (Actor, line 3), the modified handler that uses a for loop to send notifications (lines [32][33][34][35][36][37], and the message handler on the client that handles notifications (lines [15][16].…”

“…Stream programming approaches offer primitives for constructing and managing streams and can be found implemented in various paradigms: logic, functional, imperative programming. Similarly in the basic idea to RP, they use abstractions that capture and transform sequences of values, albeit with different mechanisms and constructs [36,50]. Even though they offer more explicit control of the flow of values and robust handling of changes within the system, they suffer from issues similar as in RP.…”

A Case Study on Using Functional Programming for Internet of Things Applications

“…In this article, only the code running on the base station (see Figure 3) will be considered, the code on the sensor nodes mainly handles communication with the sensor hardware and has a very simple structure since no data is stored locally. In more complex cases this part could also be implemented using functional programming techniques like functional reactive programming (Khare et al, 2015). As Figure 4 shows, the original version consists of 620 lines of source code (262 lines of C, 355 lines of ruby, 3 lines of python) plus a few shell scripts for startup tasks etc.…”