A Spatio-Temporal Data Imputation Model for Supporting Analytics at the Edge

Abstract

Current applications developed for the Internet of Things (IoT) usually involve the processing of collected data for delivering analytics and support efficient decision making. The basis for any processing mechanism is data analysis, usually having as an outcome responses in various analytics queries defined by end users or applications. However, as already noted in the respective literature, data analysis cannot be efficient when missing values are present. The research community has already proposed various missing data imputation methods paying more attention of the statistical aspect of the problem. In this paper, we study the problem and propose a method that combines machine learning and a consensus scheme. We focus on the clustering of the IoT devices assuming they observe the same phenomenon and report the collected data to the edge infrastructure. Through a sliding window approach, we try to detect IoT nodes that report similar contextual values to edge nodes and base on them to deliver the replacement value for missing data. We provide the description of our model together with results retrieved by an extensive set of simulations on top of real data. Our aim is to reveal the potentials of the proposed scheme and place it in the respective literature.

Short and Very Short Term Firm-level Load Forecasting for Warehouses: A Comparison of Machine Learning and Deep Learning Models

“…RMSE is particularly sensitive to outliers as it squares the difference between the predicted value and the observed value. RMSE presents error values in the same scale as the original variable [17] and it has been widely applied in time series analysis [18].…”

A systematic review of machine learning-based missing value imputation techniques

“…Efficient missing value imputation (Patil et al, 2010) Technique is generalized and can be utilized for many data sets (Ishay and Herman, 2015) Impute missing values and build clusters as a unified integrated process (Abdallah and Shimshoni, 2016) K-means þ radial basis function (RBF) Faster convergence speed, higher stability, accuracy (Shi et al, 2018) Local least squares Local data clustering being incorporated for improved quality and efficiency (Keerin et al, 2013) Multiple kernel density Accuracy and efficiency (Liao et al, 2018) Rough set Handles the uncertainty and vagueness existing in data sets (Amiri and Jensen, 2016) Less computational complexity (Azam et al, 2018) Overcome the problem of crispness (Raja et al, 2019) (continued ) Shell neighbor Fills in an incomplete instance in a given data set by only using its left and right nearest neighbors with respect to each factor (attribute) and generalized to deal with data sets of mixed attributes (Zhang, 2011) Sliding window Applicable for IoT devices' data (Kolomvatsos et al, 2019) Soft cluster Overcomes the problems of inconsistency (Raja and Thangavel, 2016) Decision tree Branch-exclusive splits trees (BEST) A new classification procedure that can handle missing values by using data partitioning and better accuracy (Beaulac and Rosenthal, 2020) Boosted trees Able to handle missingness from data fusion, deterministic or distribution-free data sets (D'Ambrosio et al, 2012) C4.5 Generalized approach that uses index measure in the estimation of missing values (Madhu and Rajinikanth, 2012) Classification and regression trees (CART) A robust method to deal with different missing value types (Nikfalazar et al, 2020) Decision trees and forests A higher quality of imputation using similarity and correlations…”

Embedded Data Imputation for Environmental Intelligent Sensing: A Case Study

“…In line with the former track, the work in [ 23 ] advances a double layered clustered scheme along with a consensus-based framework aimed at substituting missing values from the sensors measurements. In particular, the nodes located at the edge perform the data imputation.…”

“…Second, many existing experiments (see, e.g., [ 24 , 25 , 26 , 32 ]) are performed by simplistic data missing models, while we consider the problem of bursty missing values, which often arises when a sensor becomes unavailable for a certain (finite) period of time (a common situation for environmental sensors). Finally, compared to other works (see, e.g., [ 23 , 24 , 34 , 35 ]), which only account for one performance analysis (for instance, the imputation method accuracy), in our approach, we complement this analysis with a time assessment, which seems critical within real-time environmental data. Table 1 summarizes the main elements of novelty of our proposal with respect to some existing literature.…”