MIDIA: exploring denoising autoencoders for missing data imputation

Ma, Qian; Lee, Wang-Chien; Fu, Tao-Yang; Gu, Yu; Yu, Ge

doi:10.1007/s10618-020-00706-8

Cited by 19 publications

(5 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nazabal et al published their work on HI-VAE [60] in 2018 as a variational autoencoder-based imputation framework which can be applied for a broader set of data types under the MCAR assumption and is particularly suitable for datasets with nominal variables. Since then, novel autoencoder-based static imputation methods continue to be introduced [61]- [64]. A few of these static deep learning-based imputation methods are further summarized in Suppl.…”

Section: B Advanced Approaches For Handling Missing Data (Static Data)mentioning

confidence: 99%

Systematic Review of Advanced AI Methods for Improving Healthcare Data Quality in Post COVID-19 Era

Isgut

Gloster

Choi

et al. 2023

IEEE Rev. Biomed. Eng.

View full text Add to dashboard Cite

At the beginning of the COVID-19 pandemic, there was significant hype about the potential impact of artificial intelligence (AI) tools in combatting COVID-19 on diagnosis, prognosis, or surveillance. However, AI tools have not yet been widely successful. One of the key reason is the COVID-19 pandemic has demanded faster real-time development of AI-driven clinical and health support tools, including rapid data collection, algorithm development, validation, and deployment. However, there was not enough time for proper data quality control. Learning from the hard lessons in COVID-19, we summarize the important health data quality challenges during COVID-19 pandemic such as lack of data standardization, missing data, tabulation errors, and noise and artifact. Then we conduct a systematic investigation of computational methods that address these issues, including emerging novel advanced AI data quality control methods that achieve better data quality outcomes and, in some cases, simplify or automate the data cleaning process. We hope this article can assist healthcare community to improve health data quality going forward with novel AI development.

show abstract

Section: B Advanced Approaches For Handling Missing Data (Static Data)mentioning

confidence: 99%

Systematic Review of Advanced AI Methods for Improving Healthcare Data Quality in Post COVID-19 Era

Isgut

Gloster

Choi

et al. 2023

IEEE Rev. Biomed. Eng.

View full text Add to dashboard Cite

show abstract

“…Generally, in the AE, the latent space is determined by the distribution of the dataset. Intuitively, a sampling-based method in a latent space can be used to perform imputation of the missing element [22]- [25]. The main concern here is that the distribution of the latent space is hardly represented as a closed form, so it is inevitable for the actual imputation approximation to utilize the statistical approaches such as using the average of latent variables.…”

Section: Related Workmentioning

confidence: 99%

Anytime 3D Object Reconstruction Using Multi-Modal Variational Autoencoder

2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

For effective human-robot teaming, it is important for the robots to be able to share their visual perception with the human operators. In a harsh remote collaboration setting, data compression techniques such as autoencoder can be utilized to obtain and transmit the data in terms of latent variables in a compact form. In addition, to ensure real-time runtime performance even under unstable environments, an anytime estimation approach is desired that can reconstruct the full contents from incomplete information. In this context, we propose a method for imputation of latent variables whose elements are partially lost. To achieve the anytime property with only a few dimensions of variables, exploiting prior information of the category-level is essential. A prior distribution used in variational autoencoders is simply assumed to be isotropic Gaussian regardless of the labels of each training datapoint. This type of flattened prior makes it difficult to perform imputation from the category-level distributions. We overcome this limitation by exploiting a category-specific multi-modal prior distribution in the latent space. The missing elements of the partially transferred data can be sampled, by finding a specific modal according to the remaining elements. Since the method is designed to use partial elements for anytime estimation, it can also be applied for data over-compression. Based on the experiments on the ModelNet and Pascal3D datasets, the proposed approach shows consistently superior performance over autoencoder and variational autoencoder up to 70% data loss. The software is open source and is available from our repository 1 .

show abstract

“…The MIDA algorithm directly uses the DAE model for missing value imputation, so the improvement in imputation accuracy is limited, and the running time is too long when dealing with missing large datasets. Reference [29] explores DAE about imputation of missing data (exploring denoising autoencoders for missing data imputation MIDIA), and proposes sequential imputation of missing values MIDIA-Sequential and batch imputation of missing values MIDIA-Batch. MIDIA-Sequential trains an independent MIDIA model for each incomplete attribute, imputation missing values sequentially according to the attribute's missing rate.…”

Section: Related Workmentioning

confidence: 99%

Denoising autoencoder-based imputation of Harvard-MITx person-course dataset

Cao

Zhou²

2022

2nd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2022)

View full text Add to dashboard Cite

With the rapid development of smart education around the world, online education is becoming the new normal in the education industry. While online education is booming, a huge amount of data is generated. Online education datasets contain a large number of missing values for various reasons. The existence of missing values hinders educational research, especially educational mining and learning analysis for datasets. Based on Harvard-MITx Person-Course Dataset (HMPC) online education dataset, a batch denoising autoencoder imputation method is proposed. In order to facilitate performance evaluation, the experiment cleans the HMPC dataset, extracts some data that are not missing, and then manually introduces missing data. The experimental results show that the imputation values are similar to the results of the initial data, and the proposed algorithm better preserves the classification performance of the dataset.

show abstract

MIDIA: exploring denoising autoencoders for missing data imputation

Cited by 19 publications

References 29 publications

Systematic Review of Advanced AI Methods for Improving Healthcare Data Quality in Post COVID-19 Era

Systematic Review of Advanced AI Methods for Improving Healthcare Data Quality in Post COVID-19 Era

Anytime 3D Object Reconstruction Using Multi-Modal Variational Autoencoder

Denoising autoencoder-based imputation of Harvard-MITx person-course dataset

Contact Info

Product

Resources

About