A review and framework for the evaluation of pixel-level uncertainty estimates in satellite aerosol remote sensing

Abstract

Abstract. Recent years have seen the increasing inclusion of per-retrieval prognostic (predictive) uncertainty estimates within satellite aerosol optical depth (AOD) data sets, providing users with quantitative tools to assist in the optimal use of these data. Prognostic estimates contrast with diagnostic (i.e. relative to some external truth) ones, which are typically obtained using sensitivity and/or validation analyses. Up to now, however, the quality of these uncertainty estimates has not been routinely assessed. This study presents a review of existing prognostic and diagnostic approaches for quantifying uncertainty in satellite AOD retrievals, and it presents a general framework to evaluate them based on the expected statistical properties of ensembles of estimated uncertainties and actual retrieval errors. It is hoped that this framework will be adopted as a complement to existing AOD validation exercises; it is not restricted to AOD and can in principle be applied to other quantities for which a reference validation data set is available. This framework is then applied to assess the uncertainties provided by several satellite data sets (seven over land, five over water), which draw on methods from the empirical to sensitivity analyses to formal error propagation, at 12 Aerosol Robotic Network (AERONET) sites. The AERONET sites are divided into those for which it is expected that the techniques will perform well and those for which some complexity about the site may provide a more severe test. Overall, all techniques show some skill in that larger estimated uncertainties are generally associated with larger observed errors, although they are sometimes poorly calibrated (i.e. too small or too large in magnitude). No technique uniformly performs best. For powerful formal uncertainty propagation approaches such as optimal estimation, the results illustrate some of the difficulties in appropriate population of the covariance matrices required by the technique. When the data sets are confronted by a situation strongly counter to the retrieval forward model (e.g. potentially mixed land–water surfaces or aerosol optical properties outside the family of assumptions), some algorithms fail to provide a retrieval, while others do but with a quantitatively unreliable uncertainty estimate. The discussion suggests paths forward for the refinement of these techniques.

An AeroCom/AeroSat study: Intercomparison of Satellite AOD Datasets for Aerosol Model Evaluation

“…As the distribution is significantly wider than a Gaussian with standard deviation of 1, it appears that the observation errors are twice as large as the representation errors. See Sayer et al (2019a) for a different application of a very similar statistical test.…”

How Long Is Too Long? Variogram Analysis of AERONET Data to Aid Aerosol Validation and Intercomparison Studies

“…(2018), Sayer et al. (2020), and others in developing and refining methods to evaluate satellite data by providing a first attempt to quantify the temporal component of such uncertainties, for the global AERONET network, on time scales relevant for validation and similar analyses. It cannot provide the unknown true temporal mismatch for every given case.…”

“…One can (and should) account for this mismatch in validation exercises. This can be done, for example, when estimating RMSE of a data set from observed route mean squared deviation (which is rarely done) or when assessing compliance of a data set with either its own uncertainty estimates (Sayer et al., 2020) or GCOS goal uncertainties (GCOS, 2011). The importance of this is dependent on the characteristics of the AERONET site (with largest and most rapid mismatch uncertainties in high‐AOD conditions, and especially during biomass burning periods) as well as the error characteristics of the satellite retrieval (or model simulation) at the relevant locations and times.…”

“…(2018) and Sayer et al. (2020). This shows roughly a factor of 3 seasonal variation but is fairly insensitive to time window (within uncertainties); in part this is because σ is accounted for in the calculation, and in part this is because retrieval errors for this site are fairly large (especially in DJF for MODIS/MISR and JJA for all sensors).…”

Introducing the MISR level 2 near real-time aerosol product

“…As a result, systematic and sampling-related biases in aerosol fields are often found between model simulations and satellite observations (e.g., Buchard et al, 2015;Colarco et al, 2010;Lamarque et al, 2013;Zhang and Reid, 2009). An effective way to mitigate some of these problems is by assimilating aerosol observations into numerical models (e.g., Bocquet et al, 2015;Fu et al, 2017;Sekiyama et al, 2010;Di Tomaso et al, 2017;Werner et al, 2019;Zhang et al, 2008). Satellite observations of aerosol optical and microphysical properties are inseparable from these data assimilation activities as they offer the necessary data volume, near-global coverage, and a frequent repeat cycle.…”