A novel method for handling missing data in health care real-world study: Optimal Intact Subset Method

Abstract

Handling missing data is indispensable in health-care real-world data processing. Imputing method may introduce error and multicollinearity. Therefore, we explored (Optimal Intact Subset Method, OIS.Method) to avoid the issues. By exploring an optimal deleting way of columns and rows with missing data, a subset retaining most information of original datasets was determined. Traditionally, we can traverse all deleting ways. But the computational cost is too high to use in large datasets. OIS.Method used an indicator to determine the optimal deleting order which can ascertain the optimal deleting way and simplify computing. In order to validate the effectiveness of OIS.Method, we compared OIS.Method with five other missing data handling methods in simulated real-world classification datasets. Additionally, we validated OIS.Method in two real-world classification tasks. In simulated datasets, the performance of OIS.Method was best(highest AUC was 1). In real-world datasets, OIS.Method could acquire better classification performance. Take AUC for an example: OIS.Method VS Simple Impute VS Random Forest VS Modified Random Forest, 0.8179±0.0005 VS 0.8116±0.0002 VS 0.8087±0.0009 VS 0.8093±0.0014 in task1, and 0.7028±0.0126 VS 0.6963±0.0231 VS 0.6957±0.0247 VS 0.6699±0.0249 in task2. The calculation of OIS.Method is smaller, and it is well-suited for large real-world datasets.