Parallel Fractional Hot-Deck Imputation and Variance Estimation for Big Incomplete Data Curing

Yang, Yicheng; Kim, Jae Kwang; Cho, In Ho

doi:10.1109/tkde.2020.3029146

Cited by 9 publications

(30 citation statements)

References 39 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The mathematical symbol ' ' proposed in [1] denotes a loop that repeats a sequence of the same operation S(x) with discrete input augments. Consequently, the operations b i=a S(x i ) enumerate a sequence of S(x i ) in a vector format.…”

Section: Cell Construction and Variable Reductionmentioning

confidence: 99%

“…Meanwhile, we derive a set of the number of total donors for all recipients M = {M i | i ∈ ÃM }. [1] gave full details on unique pattern extraction and donor search, this section needs no further elaboration.…”

Section: Cell Construction and Variable Reductionmentioning

confidence: 99%

“…This paper follows the conventions in [1] to divide different types of big incomplete data according to their configurations -"big-n" data with many instances, "bigp" data with high dimensionality, and "ultra" data with concurrently many instances and high-dimensionality. None of the aforementioned existing imputation methods are adequate to cure these three categories of big incomplete data.…”

Section: Introductionmentioning

confidence: 99%

“…None of the aforementioned existing imputation methods are adequate to cure these three categories of big incomplete data. As the first attempt to handle big-n data with millions of instances or big-p data with 10, 000 variables, some authors of this paper developed an initial version of parallel-FHDI (P-FHDI) by leveraging high-performance computing (HPC) technologies and adding big dataoriented algorithms to FHDI [1]. Yet, "ultra" data was beyond the scope of the previous P-FHDI.…”

Section: Introductionmentioning

confidence: 99%

“…

…”

mentioning

confidence: 99%

See 4 more Smart Citations

Ultra Data-Oriented Parallel Fractional Hot-Deck Imputation With Efficient Linearized Variance Estimation

Yang

Kwon

Kim

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Parallel fractional hot-deck imputation (P-FHDI [1]) is a general-purpose, assumption-free tool for handling item nonresponse in big incomplete data by combining the theory of FHDI and parallel computing. FHDI cures multivariate missing data by filling each missing unit with multiple observed values (thus, hot-deck) without resorting to distributional assumptions. P-FHDI can tackle big incomplete data with millions of instances (big-n) or 10, 000 variables (big-p). However, handling ultra incomplete data (i.e., concurrently big-n and big-p) with tremendous instances and high dimensionality has posed problems to P-FHDI due to excessive memory requirement and execution time. To tackle the aforementioned challenges, we propose the ultra data-oriented P-FHDI (named UP-FHDI) capable of curing ultra incomplete data. In addition to the parallel Jackknife method, this paper enables a computationally efficient ultra data-oriented variance estimation using parallel linearization techniques. Results confirm that UP-FHDI can tackle an ultra dataset with one million instances and 10, 000 variables. This paper illustrates the special parallel algorithms of UP-FHDI and confirms its positive impact on the subsequent deep learning performance.

show abstract

Section: Cell Construction and Variable Reductionmentioning

confidence: 99%

Section: Cell Construction and Variable Reductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…

…”

mentioning

confidence: 99%

See 3 more Smart Citations

Ultra Data-Oriented Parallel Fractional Hot-Deck Imputation With Efficient Linearized Variance Estimation

Yang

Kwon

Kim

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

The impact of imputation quality on machine learning classifiers for datasets with missing values

Shadbahr,

Roberts,

Stanczuk

et al. 2023

Commun Med

View full text Add to dashboard Cite

Background Classifying samples in incomplete datasets is a common aim for machine learning practitioners, but is non-trivial. Missing data is found in most real-world datasets and these missing values are typically imputed using established methods, followed by classification of the now complete samples. The focus of the machine learning researcher is to optimise the classifier’s performance. Methods We utilise three simulated and three real-world clinical datasets with different feature types and missingness patterns. Initially, we evaluate how the downstream classifier performance depends on the choice of classifier and imputation methods. We employ ANOVA to quantitatively evaluate how the choice of missingness rate, imputation method, and classifier method influences the performance. Additionally, we compare commonly used methods for assessing imputation quality and introduce a class of discrepancy scores based on the sliced Wasserstein distance. We also assess the stability of the imputations and the interpretability of model built on the imputed data. Results The performance of the classifier is most affected by the percentage of missingness in the test data, with a considerable performance decline observed as the test missingness rate increases. We also show that the commonly used measures for assessing imputation quality tend to lead to imputed data which poorly matches the underlying data distribution, whereas our new class of discrepancy scores performs much better on this measure. Furthermore, we show that the interpretability of classifier models trained using poorly imputed data is compromised. Conclusions It is imperative to consider the quality of the imputation when performing downstream classification as the effects on the classifier can be considerable.

show abstract

Preparation of Publication Data for Mining in Scientific Review Process Management

Latypova

2024

2024 IEEE 25th International Conference of Young Professionals in Electron Devices and Materials (EDM)

View full text Add to dashboard Cite

Parallel Fractional Hot-Deck Imputation and Variance Estimation for Big Incomplete Data Curing

Cited by 9 publications

References 39 publications

Ultra Data-Oriented Parallel Fractional Hot-Deck Imputation With Efficient Linearized Variance Estimation

Ultra Data-Oriented Parallel Fractional Hot-Deck Imputation With Efficient Linearized Variance Estimation

The impact of imputation quality on machine learning classifiers for datasets with missing values

Preparation of Publication Data for Mining in Scientific Review Process Management

Contact Info

Product

Resources

About