Recommendations on benchmarks for numerical air quality model applications in China – Part 1: PM<sub>2.5</sub> and chemical species

Abstract

Abstract. Numerical air quality models (AQMs) have been applied more frequently over the past decade to address diverse scientific and regulatory issues associated with deteriorated air quality in China. Thorough evaluation of a model's ability to replicate monitored conditions (i.e., a model performance evaluation or MPE) helps to illuminate the robustness and reliability of the baseline modeling results and subsequent analyses. However, with numerous input data requirements, diverse model configurations, and the scientific evolution of the models themselves, no two AQM applications are the same and their performance results should be expected to differ. MPE procedures have been developed for Europe and North America, but there is currently no uniform set of MPE procedures and associated benchmarks for China. Here we present an extensive review of model performance for fine particulate matter (PM2.5) AQM applications to China and, from this context, propose a set of statistical benchmarks that can be used to objectively evaluate model performance for PM2.5 AQM applications in China. We compiled MPE results from 307 peer-reviewed articles published between 2006 and 2019, which applied five of the most frequently used AQMs in China. We analyze influences on the range of reported statistics from different model configurations, including modeling regions and seasons, spatial resolution of modeling grids, temporal resolution of the MPE, etc. Analysis using a random forest method shows that the choices of emission inventory, grid resolution, and aerosol- and gas-phase chemistry are the top three factors affecting model performance for PM2.5. We propose benchmarks for six frequently used evaluation metrics for AQM applications in China, including two tiers – “goals” and “criteria” – where goals represent the best model performance that a model is currently expected to achieve and criteria represent the model performance that the majority of studies can meet. Our results formed a benchmark framework for the modeling performance of PM2.5 and its chemical species in China. For instance, in order to meet the goal and criteria, the normalized mean bias (NMB) for total PM2.5 should be within 10 % and 20 %, while the normalized mean error (NME) should be within 35 % and 45 %, respectively. The goal and criteria values of correlation coefficients for evaluating hourly and daily PM2.5 are 0.70 and 0.60, respectively; corresponding values are higher when the index of agreement (IOA) is used (0.80 for goal and 0.70 for criteria). Results from this study will support the ever-growing modeling community in China by providing a more objective assessment and context for how well their results compare with previous studies and to better demonstrate the credibility and robustness of their AQM applications prior to subsequent regulatory assessments.

Tracking Daily Concentrations of PM2.5 Chemical Composition in China since 2000

“…However, a desirable total PM 2.5 simulation does not guarantee a good chemical composition simulation because small PM 2.5 simulation errors can result from the offset of positive and negative chemical component simulation errors. As shown in Figure b, although PM 2.5 chemical component concentrations correlate well with ground observations ( R = 0.57–0.71) and are in the acceptable error range (RMSE = 3.06–14.39 μg m –3 ), CMAQ underestimates SO 4 2– (NMB = −39.55%) and NH 4 + (NMB = −26.99%), while it overestimates BC (NMB = 21.14%). The simulation performance of daily mean PM 2.5 chemical component concentrations is similar but has a larger NMB for most components (Figure S7).…”

“…Although the simulation slightly underestimates (∼5%) the PM 2.5 concentrations, it is in the acceptable range considering the uncertainty of the CMAQ data set. In terms of monthly mean concentrations, the PM 2.5 samples show similar characteristics of high correlation ( R = 0.73), reasonable error (RMSE = 48.24 μg m –3 ), and mild underestimation (∼6%). However, a desirable total PM 2.5 simulation does not guarantee a good chemical composition simulation because small PM 2.5 simulation errors can result from the offset of positive and negative chemical component simulation errors.…”

Source apportionment of ambient concentration and population exposure to elemental carbon in South Korea using a three-dimensional air quality model

“…In this study, the definitions of NMB and NME follow the definitions used by Emery et al (2017). NMB and NME are performance indicators usually used in air quality model evaluations and were recommended as one of the six evaluation factors for air quality model evaluation in Huang et al (2021). In this study, the NMB and NME were determined using the simulated daily mean EC concentration and the observed daily mean EC concentration for each supersite (Eq.…”

Assessing Regional Model Predictions of Wintertime SOA from Aromatic Compounds and Monoterpenes with Precursor-specific Tracers

“…In summary, PM2.5 predictions from both inventories reasonably agree with observations. According to the benchmarks derived from regional modeling studies in China by Huang et al (2021), the MFBs of inorganic aerosols are all within the model performance criteria and most of the MFEs reach the more stringent performance goals. Although sulfate aerosols with REAS3 emission is over-predicted, it still meets the criteria of MFB < ±50% and MFE < 75% (Huang et al, 2021).…”

“…According to the benchmarks derived from regional modeling studies in China by Huang et al (2021), the MFBs of inorganic aerosols are all within the model performance criteria and most of the MFEs reach the more stringent performance goals. Although sulfate aerosols with REAS3 emission is over-predicted, it still meets the criteria of MFB < ±50% and MFE < 75% (Huang et al, 2021). The good agreement between the predicted and observed secondary inorganic aerosols suggests that the model can reproduce the oxidation capacity of the urban atmosphere in this region (Feng et al, 2021).…”