Statistical regularization for trend detection: An integrated approach for detecting long-term trends from sparse tropospheric ozone profiles

Abstract

<jats:p>Abstract. Detecting a tropospheric ozone trend from sparsely sampled ozonesonde profiles (typically once per week) is challenging due to the noise in the time series resulting from ozone's high temporal variability. To enhance trend detection we have developed a sophisticated statistical approach that utilizes a geoadditive model to assess ozone variability across a time series of vertical profiles. Treating the profile time series as a set of individual time series on discrete pressure surfaces, a class of smoothing spline ANOVA (analysis of variance) models is used for the purpose of jointly modeling multiple correlated time series (on separate pressure surfaces) by their associated seasonal and interannual variabilities. This integrated fit method filters out the unstructured noise through a statistical regularization (i.e. a roughness penalty), by taking advantage of the additional correlated data points available on the pressure surfaces above and below the surface of interest. We have applied this technique to the trend analysis of the vertically correlated time series of tropospheric ozone observations from 1) IAGOS (In-service Aircraft for a Global Observing System) commercial aircraft profiles above Europe and China, and 2) NOAA GMD's (Global Monitoring Division) ozonesonde records at Hilo, Hawaii and Trinidad Head, California. We illustrate the ability of this technique to detect a consistent trend estimate, and its effectiveness for reducing the associated uncertainty in the noisy profile data due to low sampling frequency. We also conducted a sensitivity analysis of frequent IAGOS profiles above Europe (approximately 120 profiles per month) to determine how many profiles in a month are required for reliable long-term trend detection. When ignoring the vertical correlation we found that a typical sampling strategy of 4 profiles-per-month results in 7 % of sampled trends falling outside the 2-sigma uncertainty interval derived from the full data set, with associated 10 % of mean absolute percentage error. We determined that an optimal sampling frequency is 14 profiles per month when using the integrated fit method for calculating trends; when the integrated fit method is not applied, the sampling frequency had to be increased to 18 profiles per month to achieve the same result. While our method improves trend detection from sparse data sets, the key to substantially reducing the uncertainty is to increase the sampling frequency. </jats:p>

Impact of the COVID‐19 Economic Downturn on Tropospheric Ozone Trends: An Uncertainty Weighted Data Synthesis for Quantifying Regional Anomalies Above Western North America and Europe

“…The statistical framework of trend detection for ozone vertical profile data was described by Chang et al. (2020). An important consideration of trend detection for vertically distributed time series is that the trends should not be isolated to a single narrow pressure level or layer (e.g., a depth of 10 hPa); rather we expect to observe similar trends in the neighboring pressure layers as well.…”

“…Therefore, the statistical models are applied to the normalized deviations (ND), however, the seasonal mean and SD on any pressure level can always be transformed back (to the units of ppbv) from those ND. Statistical regularization for trend and anomaly detections: The methodology developed in Chang et al. (2020) can be seen as a seasonal‐trend decomposition (Cleveland et al., 1990) designed for multiple (vertically) correlated time series. The penalized regression splines (Wood, 2006) are applied to decompose the vertical profile time series data according to their seasonal patterns and interannual changes.…”

“…Statistical regularization for trend and anomaly detections: The methodology developed in Chang et al. (2020) can be seen as a seasonal‐trend decomposition (Cleveland et al., 1990) designed for multiple (vertically) correlated time series. The penalized regression splines (Wood, 2006) are applied to decompose the vertical profile time series data according to their seasonal patterns and interannual changes.…”

“…Based on ∼45,700 ozone profiles above Europe and ∼9,900 profiles above western North America, the result is a refined estimate of the regional‐scale ozone anomalies above Europe and western North America during the COVID‐19 economic downturn, and we also quantify the impact of the 2020 ozone anomalies on the long‐term trends (1994–2020). A clear advantage of this regional‐scale approach to quantifying ozone trends is that it incorporates data from multiple sources for the identification of systematic variations; in contrast, trend detection at individual sites can be a challenge due to low signal‐to‐noise ratios (Chang et al., 2020; Saunois et al., 2012).…”

A Novel Network‐Based Approach to Determining Measurement Representation Error for Model Evaluation of Aerosol Microphysical Properties

“…Our predictors include the consecutive day (since 15 October; D ), the hour ( H ), longitude ( Lon ) and the local meteorological parameters of relative humidity ( RH ), temperature ( T ), wind speed ( Ws ), wind direction ( Wd ), and precipitation ( P ) (Equation 2) $$$E(Y)={\beta }_{0}+{f}_{1}(D)+{f}_{2}(H)+{f}_{3}(Lon)+{f}_{4}(RH)+{f}_{5}(T)+{f}_{6}(Ws)+{f}_{7}(Wd)+{f}_{8}(P)$$$ where the response is based on 1‐min data. To avoid potential overfitting resulting from a large set of predictor variables, a roughness penalty was applied to each functional approximation (so the model will not seek to capture unstructured or noisy variation in the response and the result is more generalizable; Chang et al., 2020), and the model is fitted by the generalized cross validation (GCV) criterion (Wood, 2017).…”