Characterising performance of environmental models

Bennett, Neil; Croke, Barry; Guariso, Giorgio; Guillaume, Joseph H. A.; Hamilton, Serena H.; Jakeman, Anthony J.; Marsili-Libelli, Stefano; Newham, Lachlan; Norton, J.P.; Perrin, Charles; Pierce, Suzanne A.; Robson, Barbara; Seppelt, Ralf; Voinov, Alexey; Fath, Brian D.; Andréassian, Vazken

doi:10.1016/j.envsoft.2012.09.011

Cited by 1,263 publications

(719 citation statements)

References 150 publications

Supporting

Mentioning

712

Contrasting

Unclassified

Order By: Relevance

“…Second, an exhaustive quantitative evaluation should rely on a set of metrics focussing on different aspects in order to test the ability of the model in reproducing all the important features of the system. The reader is referred to Bennett et al (2013) for a comprehensive review of techniques available for both data-division and quantitative evaluation, and to Robson (2014) for a more general assessment of environmental models." Last point -I have not published this yet -but my PMI code also takes into account the change in the uncertainty associated with the predictor variable over time.…”

Section: Reviewer #1mentioning

confidence: 99%

“…In this connection, I would like you to justify, and if pertinent expand or comment upon, your choice of evaluation metrics and methods among the ones, for example, in the recent EMS Position paper of Bennett et al (2013) on performance evaluation (they propose a 5-step procedure for evaluating the performance of models). You could also add/comment on visual methods and quantitative measures used to examine model quantities and residuals, including visual inspection.…”

mentioning

confidence: 99%

“…Second, an exhaustive quantitative evaluation should rely on a set of metrics focussing on different aspects in order to test the ability of the model in reproducing all the important features of the system. The reader is referred to Bennett et al (2013) for a comprehensive P. 29 lines 657-658: how does the framework provides a theoretical measure of computational complexity? as far as I know, this has to be computed for each IVS algorithm by considering the computation steps involved.…”

mentioning

confidence: 99%

See 2 more Smart Citations

An evaluation framework for input variable selection algorithms for environmental data-driven models

Galelli

Humphrey

Maier

et al. 2014

Environmental Modelling & Software

185

143

View full text Add to dashboard Cite

Input Variable Selection (IVS) is an essential step in data-driven modelling and is particularly relevant in environmental applications, where potential input variables are often collinear and redundant. While methods for IVS continue to emerge, each has its own advantages and limitations and no single method is best suited to all datasets and modelling purposes. Rigorous evaluation of IVS methods would allow their effectiveness to be properly identified in various circumstances. However, such evaluations are largely neglected due to the lack of guidelines to facilitate consistent and standardised assessment. This work proposes a new evaluation framework, which consists of benchmark datasets with the typical properties of environmental data, a recommended set of evaluation criteria and a website for sharing data and code. The framework is demonstrated on four IVS algorithms commonly used in environmental modelling studies. The results indicate interesting differences in the algorithms' performance that have not been identified previously.Response to Reviewers: Editor I have now received reviews of the above paper and these lead me to recommend that revision according to all the reviewers' comments is necessary. I may not send it back to reviewers, trusting that you will cut it down, otherwise few people will not bother reading it.Response to Editor comment No. 1. We significantly reduced the manuscript length by mostly focusing on Section 2 and 3, as also suggested by reviewer #2. Where possible, we also tried to reduce Section 5. Overall, we obtained a reduction of about 6 pages (from the introduction to the conclusion) with respect to the previous version of the manuscript. Furthermore, we removed Appendix A, since this material can be directly accessed from the framework website. This gives an overall reduction of 21 pages.Another issue is that I'd like it to fit better with EMS being a generic journal and so link to our key outputs. Most citations to EMS papers are to the authors themselves! Just one way to do this is to link with/refer to other key modelling concepts and issues in the journal. For example see the next paragraph.On model evaluation: that it is credible and addressed well. In this connection, I would like you to justify, and if pertinent expand or comment upon, your choice of evaluation metrics and methods among the ones, for example, in the recent EMS Position paper of Bennett et al (2013) on performance evaluation (they propose a 5-step procedure for evaluating the performance of models). You could also add/comment on visual methods and quantitative measures used to examine model quantities and residuals, including visual inspection. There are several other evaluation issues you could address/compare as well and the paper by Robson and cited below presents an excellent example in Section 13 of that paper. One of our aims for EMS is to strengthen the credibility and relevance of the modelling reported and do this whatever the environmental problem sector. That way your paper is mor...

show abstract

Section: Reviewer #1mentioning

confidence: 99%

mentioning

confidence: 99%

“…Second, an exhaustive quantitative evaluation should rely on a set of metrics focussing on different aspects in order to test the ability of the model in reproducing all the important features of the system. The reader is referred to Bennett et al (2013) for a comprehensive P. 29 lines 657-658: how does the framework provides a theoretical measure of computational complexity? as far as I know, this has to be computed for each IVS algorithm by considering the computation steps involved.…”

mentioning

confidence: 99%

See 1 more Smart Citation

An evaluation framework for input variable selection algorithms for environmental data-driven models

Galelli

Humphrey

Maier

et al. 2014

Environmental Modelling & Software

185

143

View full text Add to dashboard Cite

show abstract

“…Many model performance metrics exist, and in view of the weaknesses of individual metrics, several of these are ideally used in combination when evaluating model performance (Bennett et al, 2013). We used the coefficient of determination (r 2 ) and the mean relative absolute error in percent (referred to as MARE by Bennett et al (2013) and as RE by Arhonditsis and Brett (2004)) to evaluate the goodness of fit to observed phytoplankton biomass data for the three individual models and their ensemble mean derived from the average of daily output of phytoplankton biomass from all three models.…”

Section: Calibration and Validation Of Phytoplankton Dynamicsmentioning

confidence: 99%

Advancing projections of phytoplankton responses to climate change through ensemble modelling

Trolle

Elliott

Mooij

et al. 2014

Environmental Modelling & Software

View full text Add to dashboard Cite

A global trend of increasing health hazards associated with proliferation of toxin-producing cyanobacteria makes the ability to project phytoplankton dynamics of paramount importance.Whilst ensemble (multi-)modelling approaches have been used for a number of years to improve the robustness of weather forecasts this approach has until now never been adopted for ecosystem modelling. We show that the average simulated phytoplankton biomass derived from three different aquatic ecosystem models is generally superior to any of the three individual models in describing observed phytoplankton biomass in a typical temperate lake ecosystem, and we simulate a series of climate change projections. While this is the first multi-model ensemble approach applied for some of the most complex aquatic ecosystem models available, we consider it sets a precedent for what will become commonplace methodology in the future, as it enables increased robustness of model projections, and scenario uncertainty estimation due to differences in model structures.

show abstract

“…Evolution in model evaluation approaches, also accompanied by the creation of dedicated software tools (Fila et al 2003a, b;Tedeschi 2006;Criscuolo et al 2007;Olesen and Chang 2010), has culminated in reviews and position papers (Bellocchi et al 2010;Alexandrov et al 2011;Bennett et al 2013) with the aim of characterizing the performance of models and providing standards for publishing models in forms suitable for use by broad communities (Jakeman et al 2006;Laniak et al 2013). Several evaluation methods are available, but, usually, only a limited number of methods are used in modeling projects (as documented, for instance, by Richter et al 2012 andRitter andMuñoz-Carpena 2013), often due to time and resource constraints.…”

Section: Concepts and Toolsmentioning

confidence: 99%

Deliberative processes for comprehensive evaluation of agroecological models. A review

Bellocchi

Rivington

Matthews

et al. 2014

Agron. Sustain. Dev.

View full text Add to dashboard Cite

The use of biophysical models in agroecology has increased in the last few decades for two main reasons: the need to formalize empirical knowledge and the need to disseminate model-based decision support for decision makers (such as farmers, advisors, and policy makers). The first has encouraged the development and use of mathematical models to enhance the efficiency of field research through extrapolation beyond the limits of site, season, and management. The second reflects the increasing need (by scientists, managers, and the public) for simulation experimentation to explore options and consequences, for example, future resource use efficiency (i.e., management in sustainable intensification), impacts of and adaptation to climate change, understanding market and policy responses to shocks initiated at a biophysical level under increasing demand, and limited supply capacity. Production concerns thus dominate most model applications, but there is a notable growing emphasis on environmental, economic, and policy dimensions. Identifying effective methods of assessing model quality and performance has become a challenging but vital imperative, considering the variety of factors influencing model outputs. Understanding the requirements of stakeholders, in respect of model use, logically implies the need for their inclusion in model evaluation methods. We reviewed the use of metrics of model evaluation, with a particular emphasis on the involvement of stakeholders to expand horizons beyond conventional structured, numeric analyses. Two major topics are discussed: (1) the importance of deliberative processes for model evaluation, and (2) the role computer-aided techniques may play to integrate deliberative processes into the evaluation of agroecological models. We point out that (i) the evaluation of agroecological models can be improved through stakeholder follow-up, which is a key for the acceptability of model realizations in practice, (ii) model credibility depends not only on the outcomes of well-structured, numerically based evaluation, but also on less tangible factors that may need to be addressed using complementary deliberative processes, (iii) comprehensive evaluation of simulation models can be achieved by integrating the expectations of stakeholders via a weighting system of preferences and perception, (iv) questionnaire-based surveys can help understand the challenges posed by the deliberative process, and (v) a benefit can be obtained if model evaluation is conceived in a decisional perspective and evaluation techniques are developed at the same pace with which the models themselves are created and improved. Scientific knowledge hubs are also recognized as critical pillars to advance good modeling practice in relation to model evaluation (including access to dedicated software tools), an activity which is frequently neglected in the context of time-limited framework programs.

show abstract

Characterising performance of environmental models

Cited by 1,263 publications

References 150 publications

An evaluation framework for input variable selection algorithms for environmental data-driven models

An evaluation framework for input variable selection algorithms for environmental data-driven models

Advancing projections of phytoplankton responses to climate change through ensemble modelling

Deliberative processes for comprehensive evaluation of agroecological models. A review

Contact Info

Product

Resources

About