Scalable and Reliable Multi-dimensional Sensor Data Aggregation in Data Streaming Architectures

Abstract

Ever-increasing amounts of data and requirements to process them in real time lead to more and more analytics platforms and software systems designed according to the concept of stream processing. A common area of application is processing continuous data streams from sensors, for example, IoT devices or performance monitoring tools. In addition to analyzing pure sensor data, analyses of data for entire groups of sensors often need to be performed. Therefore, data streams of the individual sensors have to be continuously aggregated to a data stream for a group. Motivated by a real-world application scenario of analyzing power consumption in Industry 4.0 environments, we propose that such a stream aggregation approach has to allow for aggregating sensors in hierarchical groups, support multiple such hierarchies in parallel, provide reconfiguration at runtime, and preserve the scalability and reliability qualities of stream processing techniques. We propose a stream processing architecture fulfilling these requirements, which can be integrated into existing big data architectures. As all state-of-the-art stream processing frameworks have to handle a trade-off between latency, resource-efficiency, and correctness, our proposed architecture can be configured for low latency and resource-efficient computation or for always ensuring correct results. To assist adopters in choosing appropriate configuration options, we provide an experimental comparison. We present a pilot implementation of our proposed architecture and show how it is used in industry. Furthermore, in experimental evaluations we show that our solution scales linearly with the amount of sensors and provides adequate reliability in the presence of faults.

A configurable method for benchmarking scalability of cloud-native applications

“…Such computations are common to compute a seasonal trend over a long period of time (e.g., an aggregated weekly course over a period of several months). UC4 Incoming messages are aggregated to groups and groups of groups in a hierarchical fashion (Henning and Hasselbring 2020).…”

The Internet of Things and Architectures of Big Data Analytics: Challenges of Intersection at Different Domains

“…2) Data Streaming Acquisition: A suitable stream processing engine is needed to ensure the real time processing of timeliness IoT data. This module can be evaluated by measuring the response time defined as the time taken between sending a request to the server and the task completion [93]. Response time is the most appropriate evaluation metric to evaluate analytics latency and system availability [94].…”

Goals and measures for analyzing power consumption data in manufacturing enterprises

“…We suggest to organize power consumers in a hierarchical model, where groups of devices and machines are further grouped into larger groups (Henning and Hasselbring 2020). Multiple such models have to be maintained in parallel.…”

“…Furthermore, besides the focus on scalability throughout the entire Control Center architecture, an important requirement for this microservice is to reliably handle downtimes and out-of-order or late arriving measurements. Therefore, it allows to configure the required trade-off between correctness, aggregation latency, and performance (Henning and Hasselbring 2020).…”