Handling Complex Missing Data Using Random Forest Approach for an Air Quality Monitoring Dataset: A Case Study of Kuwait Environmental Data (2012 to 2018)

Abstract

In environmental research, missing data are often a challenge for statistical modeling. This paper addressed some advanced techniques to deal with missing values in a data set measuring air quality using a multiple imputation (MI) approach. MCAR, MAR, and NMAR missing data techniques are applied to the data set. Five missing data levels are considered: 5%, 10%, 20%, 30%, and 40%. The imputation method used in this paper is an iterative imputation method, missForest, which is related to the random forest approach. Air quality data sets were gathered from five monitoring stations in Kuwait, aggregated to a daily basis. Logarithm transformation was carried out for all pollutant data, in order to normalize their distributions and to minimize skewness. We found high levels of missing values for NO2 (18.4%), CO (18.5%), PM10 (57.4%), SO2 (19.0%), and O3 (18.2%) data. Climatological data (i.e., air temperature, relative humidity, wind direction, and wind speed) were used as control variables for better estimation. The results show that the MAR technique had the lowest RMSE and MAE. We conclude that MI using the missForest approach has a high level of accuracy in estimating missing values. MissForest had the lowest imputation error (RMSE and MAE) among the other imputation methods and, thus, can be considered to be appropriate for analyzing air quality data.

A Dynamic Model for Imputing Missing Medical Data: A Multiobjective Particle Swarm Optimization Algorithm

“…In the age and income example above, assume that there are missed values only of income. If the missed values are observed in a specific income range, the missed values type is MNAR [3,11]. Some types of missing data adversely affect the analysis more than other types.…”

Application of imputation methods for missing values of PM10 and O3 data: Interpolation, moving average and K-nearest neighbor methods

“…In this dataset, time and location are as observed variables. The probability of missing an observation in pollutant concentration variable is independent of other observations but dependent on time and location variables (15)(16)(17). Thus, we can conclude that the missing mechanism of PM 10 and O 3 concentrations would be MAR.…”

Handling missing data in a rheumatoid arthritis registry using random forest approach

“…Determination of whether data is MAR or MNAR is often difficult as there is no reliable technique to do so. But, in some clinical or environmental studies, 4,14,46,47 Additionally, the utilization of MI in longitudinal designs with layered data may present challenges that may need the use of MI algorithms or other approaches other than MI. [51][52][53] Another challenge is that statistical packages vary with their ease of usability in respect to the merging variables and test statistics.…”

“…To overcome this problem, Rubin suggested the theory of multiple imputation, in which missing values are imputed using the appropriate model a few times (generally 3-5 times) and a standard method is applied for the analysis. 4,9,10,11 The imputation method provides more accurate results, but problems with the application of imputation include: (a) maximum use of the available data to reduce the error for univariate data and preserve covariance in multivariate data sets; and (b) reporting the variance estimates of uncertainty caused due to the imputed value. 11 Several parametric and non-parametric techniques have been employed to deal with missing values.…”

“…These missing values often lead to misinterpretation and biased results, which could ultimately affect the overall conclusion of an investigation. [1][2][3][4] The application of statistical analyses in experiments with missing values poses serious problems, as the missing values are often automatically ignored by the statistical algorithms.…”