On model selection criteria in multimodel analysis

Abstract

[1] Hydrologic systems are open and complex, rendering them prone to multiple conceptualizations and mathematical descriptions. There has been a growing tendency to postulate several alternative hydrologic models for a site and use model selection criteria to (1) rank these models, (2) eliminate some of them, and/or (3) weigh and average predictions and statistics generated by multiple models. This has led to some debate among hydrogeologists about the merits and demerits of common model selection (also known as model discrimination or information) criteria such as AIC, AICc, BIC, and KIC and some lack of clarity about the proper interpretation and mathematical representation of each criterion. We examine the model selection literature to find that (1) all published rigorous derivations of AIC and AICc require that the (true) model having generated the observational data be in the set of candidate models; (2) though BIC and KIC were originally derived by assuming that such a model is in the set, BIC has been rederived by Cavanaugh and Neath (1999) without the need for such an assumption; and (3) KIC reduces to BIC as the number of observations becomes large relative to the number of adjustable model parameters, implying that it likewise does not require the existence of a true model in the set of alternatives. We explain why KIC is the only criterion accounting validly for the likelihood of prior parameter estimates, elucidate the unique role that the Fisher information matrix plays in KIC, and demonstrate through an example that it imbues KIC with desirable model selection properties not shared by AIC, AICc, or BIC. Our example appears to provide the first comprehensive test of how AIC, AICc, BIC, and KIC weigh and rank alternative models in light of the models' predictive performance under cross validation with real hydrologic data.

Comparison of point forecast accuracy of model averaging methods in hydrologic applications

“…Viewed as such, model averaging is a natural generalization of the more traditional aim of model selection. Indeed, the model averaging literature has its roots in the model selection literature, which continues to be a very active research area with many applications in hydrology (see e.g., Ye et al 2008). As a result of the steady increase in computer power, model averaging has gradually gained popularity as an alternative to model selection.…”

Interpretation of two-phase relative permeability curves through multiple formulations and Model Quality criteria

“…Models associated with smaller values of a given criterion are ranked higher than those associated with larger values. As shown by, e.g., Hernandez et al (2006), Ye et al (2008), andRiva et al (2011), it can be noted that KIC tends to favor models with relatively small expected information content per observation, when one considers models associated with the same number of parameters, equal minima of NLL, and the same prior probability linked to parameter values linked to NLL minimum.…”

“…In this context, a multimodel analysis based on averaging the responses of diverse models can be a powerful tool to naturally accommodate existing differences amongst models within a unique theoretical framework (Lu, 2012). Benefits of the approach have been exposed in the context of diverse environmental systems, including groundwater flow settings (e.g., Carrera and Neuman, 1996;Ye et al, 2004;Ye et al, 2008;Riva et al, 2009;Riva et al, 2011 and references therein), as well as in the interpretation of complex competitive sorption reactive processes in natural soils (Bianchi Janetti et al, 2012).…”

Complexity in microbial metabolic processes in soil nitrogen modeling: a case for model averaging

“…The BMA weights sum up to unity. BMA methods were used in various forecasting applications such as surface water hydrological forecasting (Duan et al 2007;Ajami et al 2007;Vrugt and Robinson 2007;Wöhling and Vrugt 2008;Hsu et al 2009), ground water modeling (Neuman 2003;Neuman and Wierenga 2003;Ye et al 2004;Poeter and Anderson 2005;Refsgaard et al 2007;Ye et al 2008;Rojas et al 2009) and weather forecasting (Raftery et al 2003(Raftery et al , 2005 and have shown promising results in dealing with model predictive uncertainty.…”