Performance Evaluation of Imputation Methods for Incomplete Datasets

Yenduri, Sumanth; Iyengar, S. Sitharama

doi:10.1142/s0218194007003173

Cited by 19 publications

(11 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several studies showed that NN may be superior over other hot-deck methods even though results may be dependent from the choice of the metric used to gauge the similarity or the dissimilarity of recipients to donors [7]. …”

Section: Introductionmentioning

confidence: 99%

Nearest neighbor imputation algorithms: a critical evaluation

Beretta

Santaniello

2016

BMC Med Inform Decis Mak

455

305

View full text Add to dashboard Cite

BackgroundNearest neighbor (NN) imputation algorithms are efficient methods to fill in missing data where each missing value on some records is replaced by a value obtained from related cases in the whole set of records. Besides the capability to substitute the missing data with plausible values that are as close as possible to the true value, imputation algorithms should preserve the original data structure and avoid to distort the distribution of the imputed variable. Despite the efficiency of NN algorithms little is known about the effect of these methods on data structure.MethodsSimulation on synthetic datasets with different patterns and degrees of missingness were conducted to evaluate the performance of NN with one single neighbor (1NN) and with k neighbors without (kNN) or with weighting (wkNN) in the context of different learning frameworks: plain set, reduced set after ReliefF filtering, bagging, random choice of attributes, bagging combined with random choice of attributes (Random-Forest-like method).ResultsWhatever the framework, kNN usually outperformed 1NN in terms of precision of imputation and reduced errors in inferential statistics, 1NN was however the only method capable of preserving the data structure and data were distorted even when small values of k neighbors were considered; distortion was more severe for resampling schemas.ConclusionsThe use of three neighbors in conjunction with ReliefF seems to provide the best trade-off between imputation error and preservation of the data structure. The very same conclusions can be drawn when imputation experiments were conducted on the single proton emission computed tomography (SPECTF) heart dataset after introduction of missing data completely at random.

show abstract

Section: Introductionmentioning

confidence: 99%

Nearest neighbor imputation algorithms: a critical evaluation

Beretta

Santaniello

2016

BMC Med Inform Decis Mak

455

305

View full text Add to dashboard Cite

show abstract

“…1) results may be dependent from the choice of the metric used to gauge the similarity (or the distance) between observations [45]. 2) due to the knn search, when the dataset has many samples, the algorithm might be time consuming.…”

Section: ) Missing Data Imputation Methodsmentioning

confidence: 99%

Human Digital Twin for Fitness Management

et al. 2020

View full text Add to dashboard Cite

Our research work describes a team of human Digital Twins (DTs), each tracking fitness-related measurements describing an athlete's behavior in consecutive days (e.g. food income, activity, sleep). After collecting enough measurements, the DT firstly predicts the physical twin performance during training and, in case of non-optimal result, it suggests modifications in the athlete's behavior. The athlete's team is integrated into SmartFit, a software framework for supporting trainers and coaches in monitoring and manage athletes' fitness activity and results. Through IoT sensors embedded in wearable devices and applications for manual logging (e.g. mood, food income), SmartFit continuously captures measurements, initially treated as the dynamic data describing the current physical twins' status. Dynamic data allows adapting each DT's status and triggering the DT's predictions and suggestions. The analyzed measurements are stored as the historical data, further processed by the DT to update (increase) its knowledge and ability to provide reliable predictions. Results show that, thanks to the team of DTs, SmartFit computes trustable predictions of the physical twins' conditions and produces understandable suggestions which can be used by trainers to trigger optimization actions in the athletes' behavior. Though applied in the sport context, SmartFit can be easily adapted to other monitoring tasks. INDEX TERMS Counterfactual explanations, digital twins, Internet of Things, machine learning, smart health, sociotechnical design, wearables.

show abstract

“…In the calculation of ( ), Ŷ and 2 denote the pilot estimates of fitted values and the variance of the error terms for chosen a pilot smoothing parameter ( ), respectively. In practice, since the variance 2 is generally unknown, 2 is used and it can be easily calculated by the equation (11). Note also that variance of error terms ( 2 ) is called as a variance of the regression model.…”

Section: Selection Of the Smoothing Parametermentioning

confidence: 99%

“…In the literature, some examples of imputation include Schafer, Batista and Monard, Rubin and van der Laan, Yenduri and Iyengar and Andridge and Little. [8][9][10][11][12] There are also various imputation methods used for different types of data such as fuzzy K-means, singular value decomposition, multiple imputations by chained equations. 13,14 Most of these methods are developed to solve missing data problems but some of these methods are also suitable for solving the right-censorship problems.…”

mentioning

confidence: 99%

Estimation of Right Censored Nonparametric Regression Solved by kNN Imputation: A Comparative Study

Yılmaz¹,

Aydın²

2019

Turkiye Klinikleri J Biostat

View full text Add to dashboard Cite

This paper introduces an estimation procedure for the right-censored nonparametric regression model using smoothing spline method. In this process, to overcome the censorship problem we used an imputation method based on k-nearest neighbors (kNN). Among some known censorship solutions, such as Kaplan-Meier weights (Kaplan and Meier, Miller) and Synthetic data transformation (Koul et al.), the most important advantage of the kNN imputation method is that it does not depend on a distribution. After solving the problem of censorship, the most important problem in obtaining the optimal estimation of non1parametric regression function by using smoothing spline will be the selection of the smoothing parameter. In order to achieve this aim, three commonly used criteria such as generalized cross-validation (GCV), Bayesian information criterion (BIC) and risk estimation using classical pilots (RECP) are considered in this study. A Monte-Carlo simulation study and a "kidney infection recurrence" data are carried out to realize the purposes of this study. Thus, it is determined that which selection criterion is more successful in estimating the non-parametric model with right censored data. Obtained results from both simulation and real-world studies show that BIC has remarkable performance among others. Also, it can be seen that GCV is better than BIC for large sample size. RECP has mediocre performance.

show abstract

Performance Evaluation of Imputation Methods for Incomplete Datasets

Cited by 19 publications

References 17 publications

Nearest neighbor imputation algorithms: a critical evaluation

Nearest neighbor imputation algorithms: a critical evaluation

Human Digital Twin for Fitness Management

Estimation of Right Censored Nonparametric Regression Solved by kNN Imputation: A Comparative Study

Contact Info

Product

Resources

About