A Framework for Real-Time Information Derivation from Big Sensor Data

Abstract

In data-intensive real-time applications, e.g., transportation management and location-based services, the amount of sensor data is exploding. In these applications, it is desirable to extract value-added information, e.g., fast driving routes, from sensor data streams in real-time rather than overloading users with massive raw data. However, achieving the objective is challenging due to the data volume and complex data analysis tasks with stringent timing constraints. Most existing big data management systems, e.g., Hadoop, are not directly applicable to real-time sensor data analytics, since they are timing agnostic and focus on batch processing of previously stored data that are potentially outdated and subject to I/O overheads. To address the problem, we design a new real-time big data management framework, which supports a non-preemptive periodic task model for continuous in-memory sensor data analysis and a schedulability test based on the EDF (Earliest Deadline First) algorithm to derive information from current sensor data in realtime by extending the map-reduce model originated in functional programming. As a proof-of-concept case study, a prototype system is implemented. In the performance evaluation, it is empirically shown that all deadlines can be met for the tested sensor data analysis benchmarks.

Real-Time Information Derivation from Big Sensor Data via Edge Computing

“…An initial result of this work was published in a conference [21]. This paper significantly extends [21] as follows.…”

“…This paper significantly extends [21] as follows. We support adaptive sensor data transfer from IoT devices to the edge server, which performs real-time analysis of the sensor data, based on data importance.…”

“…We support adaptive sensor data transfer from IoT devices to the edge server, which performs real-time analysis of the sensor data, based on data importance. (In the conference paper [21], data transfer issues were not considered.) We have extended the system architecture to support efficient sensor data transfer as discussed in Section 2.…”

High Performance Priority Packets Scheduling Mechanism for Big Data in Smart Cities

“…All of these algorithms focus on how to ensure high scheduling network efficiency and appropriate allocation of network resources. For example, scheduling algorithms such as Earliest deadline First (EDF) [34], or First Come First Served (FCFS) [35], use data packets for differentiation between priority and nonpriority packets [36], while other studies proposed algorithms that give the priority packets a higher permission for single node transmission that preempt packets with low priority [37]. Quality of Service (QoS) approaches have been used extensively as scheduling methods in big data smart cities.…”