Embedded Data Imputation for Environmental Intelligent Sensing: A Case Study

Abstract

Recent developments in cloud computing and the Internet of Things have enabled smart environments, in terms of both monitoring and actuation. Unfortunately, this often results in unsustainable cloud-based solutions, whereby, in the interest of simplicity, a wealth of raw (unprocessed) data are pushed from sensor nodes to the cloud. Herein, we advocate the use of machine learning at sensor nodes to perform essential data-cleaning operations, to avoid the transmission of corrupted (often unusable) data to the cloud. Starting from a public pollution dataset, we investigate how two machine learning techniques (kNN and missForest) may be embedded on Raspberry Pi to perform data imputation, without impacting the data collection process. Our experimental results demonstrate the accuracy and computational efficiency of edge-learning methods for filling in missing data values in corrupted data series. We find that kNN and missForest correctly impute up to 40% of randomly distributed missing values, with a density distribution of values that is indistinguishable from the benchmark. We also show a trade-off analysis for the case of bursty missing values, with recoverable blocks of up to 100 samples. Computation times are shorter than sampling periods, allowing for data imputation at the edge in a timely manner.

PV Forecasting Model Development and Impact Assessment via Imputation of Missing PV Power Data

“…In the KNN imputation method, the results of calculations using distance measurement formulas, e.g., Euclidean distance, are employed for replacement. Missing data are replaced using the values of the k-nearest neighbors [4]. The KNN algorithm is generally recognized as suitable for imputing missing values in time-series data, such as power consumption data [5].…”

Imputation of missing microclimate data of coffee-pine agroforestry with machine learning

“…Additionally, it is important to consider the temporal components in the data, as linear methods often neglect these components [17]. Furthermore, the application of machine learning methodologies, including deep learning and ensemble learning, has been suggested for the imputation or estimation of missing data, offering potential advantages in this context [18], [19]. Moreover, the application of machine learning methods, such as support vector machines and random forests, has been successful in spatial interpolation, which can be relevant for addressing missing microclimate data [20].…”