Statistical Analysis of Particulate Matter Data in Doha, Qatar

Abstract

Pollution in Doha is measured using passive, active and automatic sampling. In this paper we consider data automatically sampled in which various pollutants were continually collected and analysed every hour. At each station the sample is analysed on-line and in real time and the data is stored within the analyser, or a separate logger so it can be downloaded remotely by a modem. The accuracy produced enables pollution episodes to be analysed in detail and related to traffic flows, meteorology and other variables. Data has been collected hourly over more than 6 years at 3 different locations, with measurements available for various pollutants-for example, ozone, nitrogen oxides, sulphur dioxide, carbon monoxide, THC, methane and particulate matter (PM1.0, PM2.5 and PM10), as well as meteorological data such as humidity, temperature, and wind speed and direction. Despite much care in the data collection process, the resultant data has long stretches of missing values, when the equipment has malfunctioned-often as a result of more extreme conditions. Our analysis is twofold. Firstly, we consider ways to "clean" the data, by imputing missing values, including identified outliers. The second aspect specifically considers prediction of each particulate (PM1.0, PM2.5 and PM10) 24 hours ahead, using current (and previous) pollution and meteorological data. In this case, we use vector autoregressive models, compare with decision trees and propose variable selection criteria which explicitly adapt to missing data. Our results show that the regression tree models, with no variable transformations, perform the best, and that attempts to impute missing values are hampered by nonrandom missingness.

Investigation of PM10 prediction utilizing data mining techniques: Analyze by topic

“…Data is often transformed in order to improve their further processing. Physical vectors, such as wind speed (WS) and the wind direction (WD) were transformed to u and v components (Huebnerova & Michalek, 2014;Vlachogiannis & Sfetsos, 2006) and into two separate data by sine and cosine due to their discontinuity (Grivas & Chaloulakou, 2006; S. Lira et al, 2007;Paschalidou et al, 2011;Taylor et al, 2018). In addition, the wind direction index (WDI) transforms WD by a specific equation and proposed by two different methods.…”

“…Data transformation of both STV and LTV were used differently depending on research articles (McKendry, 2002;Chaloulakou et al, 2003;Kukkonen, 2003;Aldrin & Haff, 2005;Hooyberghs et al, 2005;Grivas & Chaloulakou, 2006;Papanastasiou et al, 2007;S. Lira et al, 2007;Stadlober et al, 2008;Barai et al, 2009;Hrust et al, 2009;Paschalidou et al, 2011;Fernando et al, 2012;Nejadkoorki & Baroutian, 2012;Huebnerova & Michalek, 2014;Alam & McNabola, 2015;Dedovic et al, 2016;Dotse et al, 2018;Faganeli Pucer et al, 2018;Taylor et al, 2018).…”

“…Another transformation technique was the logarithm transformation which was applied to various input features (Aldrin & Haff, 2005;Díaz-Robles et al, 2008;Taylor et al, 2018). In addition, the output of the model, especially the Multiple linear regression model, was applied to the Box-Cox transformation before the model creation process to reduce skewness of data.…”

“…The forward selection was used to find a subset of the best features by adding one by one feature to the model and select only the features that pass a threshold (Lesar & Filipči c, 2017;Martínez et al, 2018;Sfetsos & Vlachogiannis, 2009). The backward elimination searched for the best subset of features by adding all input features to the model and respectively eliminate disqualified features (Cai et al, 2009;Martínez et al, 2018;Taylor et al, 2018). The stepwise regression was a combination of the forward selection and the backward elimination.…”

“…However, it is noted that they are less frequently used than regression analysis and ANN. This included the decision tree (DT) with various enhancement techniques, that is, the classification and regression tree (CART), the boosted regression tree (BRT), and the random forest (Faganeli Pucer et al, 2018;Martínez et al, 2018;Sayegh et al, 2014;Slini et al, 2006;Stoimenova et al, 2017;Taylor et al, 2018).…”