An energy-aware data cleaning workflow for real-time stream processing in the internet of things

Abstract

The Internet of things (IoT) has recently transformed the internet, enabling the communication between every kind of objects (things). The growing number of sensors and smart devices enhanced data creation and collection capabilities and led to an explosion of generated data in the form of Data Streams. Processing these data streams is complex and presents challenges and opportunities in the stream processing field. Due to the inherent lacking of accuracy and completeness of sensor generated data, the quality of raw data is often poor. Data cleaning tasks are required to help increasing the quality of the data being processed in an IoT application. This work proposes a data stream processing workflow for IoT to be deployed at the edge of the network. It performs a fast data cleaning with low power consumption from edge and sensor nodes. The edge computing paradigm is used to bring the data cleaning task closer to the data sources and allow actions to be triggered immediately. In addition, an energy-aware data collection component is designed to reduce the network traffic and, as a consequence, decrease the power consumption of the network devices. The proposed workflow enables the deployment of long running real-time processing systems on remote outdoor environments.

Middleware Supporting PIS: Requirements, Solutions, and Challenges

“…In the logistic chain traceability system example, to achieve energy efficiency, the PIS middleware could: (i) at design time, choose the most energy-efficient consensus algorithm for sharing securely and transparently data between the stakeholders (Sedlmeir et al, 2020); (ii) at runtime, reduce the volume of exchanged data by filtering data based on their content or minimizing the frequency of data transmissions (de Oliveira et al, 2020). For energy-awareness, the middleware could adapt the frequency of data transmissions to keep energy consumption above a certain level of energy budget or transmit energy consumption information to the applica-tion level, for example, to inform the end-user about the consumption of the energy budget.…”

“…These techniques dynamically reconfigure rates and filter thresholds to trade-off data quality against resource utilization. de Oliveira et al (2020) proposed a data stream processing workflow to be deployed at the network's edge to perform data cleaning tasks.…”