Comparison of point forecast accuracy of model averaging methods in hydrologic applications

Abstract

Multi-model averaging is currently receiving a surge of attention in the atmospheric, hydrologic, and statistical literature to explicitly handle conceptual model uncertainty in the analysis of environmental systems and derive predictive distributions of model output. Such density forecasts are necessary to help analyze which parts of the model are well resolved, and which parts are subject to considerable uncertainty. Yet, accurate point predictors are still desired in many practical applications. In this paper, we compare a suite of different model averaging techniques by their ability to improve forecast accuracy of environmental systems. We compare equal weights averaging (EWA), Bates-Granger model averaging (BGA), averaging using Akaike's information criterion (AICA), and Bayes' Information Criterion (BICA), Bayesian model averaging (BMA), Mallows model averaging (MMA), and Granger-Ramanathan averaging (GRA) for two different hydrologic systems involving water flow through a 1950 km 2 watershed and 5 m deep vadose zone. Averaging methods with weights restricted to the multi-dimensional simplex (positive weights summing up to one) are shown to have considerably larger forecast errors than approaches with unconstrained weights. Whereas various sophisticated model averaging approaches have recently emerged in the literature, our results convincingly demonstrate the advantages of GRA for hydrologic applications. This method achieves similar performance as MMA and BMA, but is much simpler to implement and use, and computationally much less demanding.

Technical note: Combining quantile forecasts and predictive distributions of streamflows

“…In Abrahart and See (2002) different combination methods for hydrological forecast models are compared. Diks and Vrugt (2010) compare different model averaging approaches, showing that a simple regression method could result in improvements comparable to more sophisticated methods.…”

Assessment of hydrological model predictive ability given multiple conceptual geological models

“…Bayesian model averaging (BMA) [Hoeting et al, 1999;Neuman, 2003;Rojas et al, 2008] is a theoretically comprehensive and computationally demanding framework that not only produces optimal predictions, but also allows estimation of the total prediction uncertainty. Less sophisticated model averaging techniques, such as equal weights averaging (EWA), Bates-Granger averaging (BGA), and Granger-Ramanathan averaging (GRA) have in some cases turned out to be as good as the computationally much more demanding BMA method [Diks and Vrugt, 2010]. In the case of BMA, the model weights are calculated based on a combination of prior belief and the model performance during the calibration, while only the model performance is included for the simpler averaging techniques such as BGA and GRA.…”

“…In many scientific disciplines the use of multimodels and model averaging based on a weighting of the individual models has therefore become increasingly popular, as it will often produce robust predictions compared to a single model prediction [Cavadias and Morin, 1986;Poeter and Anderson, 2005;Sánchez et al, 2009;Winter and Nychka, 2010;Diks and Vrugt, 2010]. Bayesian model averaging (BMA) [Hoeting et al, 1999;Neuman, 2003;Rojas et al, 2008] is a theoretically comprehensive and computationally demanding framework that not only produces optimal predictions, but also allows estimation of the total prediction uncertainty.…”

Sequential data assimilation with multiple nonlinear models and applications to subsurface flow

“…. , u i M (n) with data d, using (7) and (8), obtain N a realisations of "analyzed state", w i (n), i = 1, . .…”