Combining Fourier and lagged k -nearest neighbor imputation for biomedical time series data

Abstract

Most clinical and biomedical data contain missing values. A patient’s record may be split across multiple institutions, devices may fail, and sensors may not be worn at all times. While these missing values are often ignored, this can lead to bias and error when the data are mined. Further, the data are not simply missing at random. Instead the measurement of a variable such as blood glucose may depend on its prior values as well as that of other variables. These dependencies exist across time as well, but current methods have yet to incorporate these temporal relationships as well as multiple types of missingness. To address this, we propose an imputation method (FLk-NN) that incorporates time lagged correlations both within and across variables by combining two imputation methods, based on an extension to k-NN and the Fourier transform. This enables imputation of missing values even when all data at a time point is missing and when there are different types of missingness both within and across variables. In comparison to other approaches on three biological datasets (simulated and actual Type 1 diabetes datasets, and multi-modality neurological ICU monitoring) the proposed method has the highest imputation accuracy. This was true for up to half the data being missing and when consecutive missing values are a significant fraction of the overall time series length.

A New Fuzzy Logic-Based Similarity Measure Applied to Large Gap Imputation for Uncorrelated Multivariate Time Series

“…Up to now, numerous successful researches have been devoted to complete missing data in multivariate time series imputation such as [10,11,[20][21][22][23][24][25][26][27][28]. Imputation techniques can be categorized in different perspectives: model-based or machine learningbased and clustering-based imputation techniques.…”

“…The results showed that the combination of MICE and RF was more efficient than original methods for multivariate imputation. K-Nearest Neighbors ( -NN)-based imputation is also a popular method for completing missing values such as [11,26,27,[30][31][32]. This approach identifies most similar patterns in the space of available features to impute missing data.…”

“…Save position of ( ) to (25) end if (26) ← + (27) end while (28) return position of -the most similar window to having the maximum fuzzy similarity measure in the list. (29) end for (30) for data do (31) Perform…”

“…The analysis of these data with remarkable size and shape allows sea biologists to reveal events such as algal blooms, understand phytoplankton processes [3] in detail, or detect sea pollution and so on. But the data have a lot of missing values: 62.2% for phosphate, 59.9% for nitrate, 27.22% for pH, etc., and the size of missing data varies from one-third hour to several months.…”

Dynamic time warping-based imputation for univariate time series data

“…Concerning the imputation task for multivariate time series, many studies have been investigated using machine learning techniques as Shah et al (2014), Liao et al (2014), Rahman et al (2015) and model techniques such as Raghunathan and Siscovick (1996), Schafer (1997), Van Buuren et al (1999), Raghunathan et al (2001), Royston (2007), Joseph et al (2009), Stuart et al (2009, Lee and Carlin (2010), Spratt et al (2010), Gelman et al (2015, Deng et al (2016). The efficiency of these algorithms is based on correlations between signals or their features, and missing values are estimated from the observed values.…”