Imputation of Missing Values in Time Series with Lagged Correlations

Abstract

Missing values are a common problem in real world data and are particularly prevalent in biomedical time series, where a patient's medical record may be split across multiple institutions or a device may briefly fail. These data are not missing completely at random, so ignoring the missing values can lead to bias and error during data mining. However, current methods for imputing missing values have yet to account for the fact that variables are correlated and that those relationships exist across time. To address this, we propose an imputation method (FLk-NN) that incorporates time lagged correlations both within and across variables by combining two imputation methods, based on an extension to k-NN and the Fourier transform. This enables imputation of missing values even when all data at a time point is missing and when there are different types of missingness both within and across variables. In comparison to other approaches on two biological datasets (simulated glucose in Type 1 diabetes and multi-modality neurological ICU monitoring) the proposed method has the highest imputation accuracy. This was true for up to half the data being missing and when consecutive missing values are a significant fraction of the overall time series length.

Imputing the Missing Values in Iot Using Estcp Model

“…Since observations taken at time 't' are likely to be correlated with observations taken at times 't−1', than observations at t−2, t-3, t-4, t-5 and so on [1]. If data including lagged variables of 1 time step (t−1) is missing then , data including consecutive lagged variables up to 4 time steps (t−2,t−3 andt−4,t-5) will be taken and mean will be found.An extension to STCP with time lagged correlations has been proposed in this paper with the assumption that correlations may persist for a period of time in a sensor.…”

“…The sensed data were stored as time-series data and the samples were taken from the related time-series data to appraise missing values from the spatial and temporal dimensions to which weights were assigned. Simulations performed on the datasets taken, overtook the existing imputation methods in terms of accuracy.Shah Atiqur Rahman et al [1] proposed an imputation technique called fuzzy logic k nearest neighbour that used time lagged correlations to impute missing data. The proposed technique is the combination of two other imputation techniques viz.…”

Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices

“…The K-Nearest Neighbor (k-NN) method is one of the most used methods for data imputation of data acquired from motion, and magnetic/mechanical sensors [ 52 , 53 , 54 , 55 ]. The k-NN method has several variants that can be used for data imputation, such as MKNNimpute (K-nearest neighbor imputation method based on Mahalanobis distance), SKNNimpute (sequential K-nearest neighbor method-based imputation), and KNNimpute (K-nearest neighbor imputation) [ 52 , 53 ].…”

“…There are other methods related to data imputation, including multiple imputation [ 58 ], hot/cold imputation [ 59 ], maximum likelihood [ 60 ], Bayesian estimation [ 60 ], expectation maximization [ 54 , 61 , 62 ], discarding instances [ 18 ], pairwise deletion [ 18 ], unconditional mean imputation [ 18 ], conditional mean imputation [ 18 ], hot deck imputation [ 18 ], cold deck imputation [ 18 ], substitution method [ 18 ], linear regression [ 18 ], logistic regression [ 18 ], expectation-maximization (EM) algorithm [ 18 ], probabilistic neural networks [ 18 ], fuzzy min–max neural networks [ 18 ], general regression auto associative neural network [ 18 ], tree-based methods [ 18 ], multi-matrices factorization model (MMF) [ 63 ], mean imputation (MEI) [ 54 , 62 ], Multivariate Imputation by Chained Equations (MICE) [ 54 , 62 ], Fourier method [ 62 ], and Fourier and lagged k-NN combined system (FLk-NN) [ 54 , 62 , 64 ].…”

“…As the data imputation methods should be able to impute the empty instances of the data acquired by motion and magnetic/mechanical sensors, the methods that are able to be used with this purpose are MEI, EM, MICE, and FLk-NN [ 54 ]. However, k-NN can be applied with the comparison between the history of the data acquisition that is similar to the data acquired in the stream with faulty values [ 54 ]. It emerges from the reviewed literature that data imputation may be avoided for acoustic and location sensors, because of the slow variability of their signals.…”