A diagnostic suite to assess NWP performance

Abstract

[1] A suite of numerical weather prediction (NWP) verification diagnostics applicable to both scalar and vector variables is developed, highlighting the normalization and successive decomposition of model errors. The normalized root-mean square error (NRMSE) is broken down into contributions from the normalized bias (NBias) and the normalized pattern error (NPE). The square of NPE, or the normalized error variance a, is further analyzed into phase and amplitude errors, measured respectively by the correlation and the variance similarity. The variance similarity diagnostic is introduced to verify variability e.g. under different climates. While centered RMSE can be reduced by under-prediction of variability in the model, a penalizes over-and under-prediction of variability equally. The error decomposition diagram, the correlation-similarity diagram and the anisotropy diagram are introduced. The correlation-similarity diagram was compared with the Taylor diagram: it has the advantage of analyzing the normalized error variance geometrically into contributions from the correlation and variance similarity. Normalization of the error metrics removes the dependence on the inherent variability of a variable and allows comparison among quantities of different physical units and from different regions and seasons. This method was used to assess the Coupled Ocean/Atmospheric Mesoscale Prediction System (COAMPS). The NWP performance degrades progressively from the midlatitudes through the sub-tropics to the tropics. But similar cold and moist biases are noted and position and timing errors are the main cause of pattern errors. Although the suite of metrics is applied to NWP verification here, it is generally applicable as diagnostics for differences between two data sets.

High-resolution dynamical downscaling of re-analysis data over the Kerguelen Islands using the WRF model

“…The model output is first evaluated against station data, as given by the NOAA GSOD dataset, and radiosonde data, given by NOAA IGRA, for the Port-aux-Français station, the only location in the island for which observational data is available. The diagnostic suite proposed by Koh et al (2012) is employed to assess the model performance. With respect to station data, the main model biases are a lower daily-mean temperature by about 1 to 2 K (mostly due to lower maximum temperatures suggesting excessive low-level cloud cover in the model), heavier precipitation (in particular in the winter season) and weaker daily-mean wind speeds (larger biases in the summer season with a magnitude of up to~2 ms −1 ).…”

“…A random forecast has α = 1 so that α < 1 is recommended. More information about these diagnostics can be found in Koh et al (2012).…”

“…The normalised bias is given by the bias divided by the standard deviation of the discrepancy between the model and observations. This diagnostic can be used to determine whether the model biases are significant: as shown in Koh et al (2012), if |μ| < 0.5 the contribution of the bias to the root mean square error is less than~10% and the biases can be considered not significant when compared to the error variance. The correlation is a measure of the phase agreement between the modelled and observed signal variations.…”

“…The WRF performance is assessed using the diagnostic suite proposed by Koh et al (2012) which includes the model bias, normalised bias (μ), correlation (ρ), variance similarity (η) and normalised error variance (α). The bias is defined as the mean discrepancy between the model and observations.…”

Scale-Selective Validation of Morphodynamic Models

“…which, with q =1, is the form as used by Wang et al (2004) and Koh et al (2012) and named contrast measure and variance similarity, respectively. In morphodynamic modelling, where the predictand is the bathymetry, the interpretation of ρ po and ˆp o     in terms of bed features is far from trivial, since multiple scales are generally present in the observed and computed bathymetry ( Fig.…”