Self–Training With Quantile Errors for Multivariate Missing Data Imputation for Regression Problems in Electronic Medical Records: Algorithm Development Study

Abstract

Background When using machine learning in the real world, the missing value problem is the first problem encountered. Methods to impute this missing value include statistical methods such as mean, expectation-maximization, and multiple imputations by chained equations (MICE) as well as machine learning methods such as multilayer perceptron, k-nearest neighbor, and decision tree. Objective The objective of this study was to impute numeric medical data such as physical data and laboratory data. We aimed to effectively impute data using a progressive method called self-training in the medical field where training data are scarce. Methods In this paper, we propose a self-training method that gradually increases the available data. Models trained with complete data predict the missing values in incomplete data. Among the incomplete data, the data in which the missing value is validly predicted are incorporated into the complete data. Using the predicted value as the actual value is called pseudolabeling. This process is repeated until the condition is satisfied. The most important part of this process is how to evaluate the accuracy of pseudolabels. They can be evaluated by observing the effect of the pseudolabeled data on the performance of the model. Results In self-training using random forest (RF), mean squared error was up to 12% lower than pure RF, and the Pearson correlation coefficient was 0.1% higher. This difference was confirmed statistically. In the Friedman test performed on MICE and RF, self-training showed a P value between .003 and .02. A Wilcoxon signed-rank test performed on the mean imputation showed the lowest possible P value, 3.05e-5, in all situations. Conclusions Self-training showed significant results in comparing the predicted values and actual values, but it needs to be verified in an actual machine learning system. And self-training has the potential to improve performance according to the pseudolabel evaluation method, which will be the main subject of our future research.

Comparing human milk macronutrients measured using analyzers based on mid-infrared spectroscopy and ultrasound and the application of machine learning in data fitting

“…model_selection.train_test_split (X, Y, test_size = 0.20, random_state = 1), X refers to ultrasonic, Y refers to MIR, and then sklearn.linear_model.LinearRegression (scikitlearn 0.24.2) was used for model training. We randomly selected 80% of the dataset to be used in generating the working model (called "training data") and the other 20% to test the model (called "test data"), as recommended in previous literature [41][42][43]. These steps were used in creating the correction models for protein, fat, lactose, and energy content.…”

Al-Biruni Earth Radius Optimization Based Algorithm for Improving Prediction of Hybrid Solar Desalination System

“…Contrarily, in experimental approaches, expensive and time-consuming trials are conducted out before system characteristics and the considered results are correlated statistically. Unfortunately, both approaches' outcomes may be impacted by noisy circumstances [36]. Therefore, it would be ideal to create a reliable and accurate approach for simulating and forecasting the effects design and operational parameters on the HDH-SSF's performance.…”