Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module

Abstract

Abstract. Hindcasting experiments (conducting a model forecast for a time period in which observational data are available) are being undertaken increasingly often by the integrated assessment model (IAM) community, across many scales of models. When they are undertaken, the results are often evaluated using global aggregates or otherwise highly aggregated skill scores that mask deficiencies. We select a set of deviation-based measures that can be applied on different spatial scales (regional versus global) to make evaluating the large number of variable-region combinations in IAMs more tractable. We also identify performance benchmarks for these measures, based on the statistics of the observational dataset, that allow a model to be evaluated in absolute terms rather than relative to the performance of other models at similar tasks. An ideal evaluation method for hindcast experiments in IAMs would feature both absolute measures for evaluation of a single experiment for a single model and relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models, such as in community intercomparison studies. The performance benchmarks highlight the use of this scheme for model evaluation in absolute terms, providing information about the reasons a model may perform poorly on a given measure and therefore identifying opportunities for improvement. To demonstrate the use of and types of results possible with the evaluation method, the measures are applied to the results of a past hindcast experiment focusing on land allocation in the Global Change Assessment Model (GCAM) version 3.0. The question of how to more holistically evaluate models as complex as IAMs is an area for future research. We find quantitative evidence that global aggregates alone are not sufficient for evaluating IAMs that require global supply to equal global demand at each time period, such as GCAM. The results of this work indicate it is unlikely that a single evaluation measure for all variables in an IAM exists, and therefore sector-by-sector evaluation may be necessary.

Spatial Sequential Modeling and Predication of Global Land Use and Land Cover Changes by Integrating a Global Change Assessment Model and Cellular Automata

“…In the AgLU component of GCAM, changes in the 283 world regions are modeled based on a range of drivers, including population growth, income, technology improvements, crop productivity, labor costs, energy demand, and environmental policies (Snyder et al, ; Vittorio et al, ). As an integrated assessment model, the land use modeling in GCAM considers comprehensive land use types.…”

A crop yield change emulator for use in GCAM and similar models: Persephone v1.0

“…-The 99 emulated yields returned by the mean response function are compared to the mean yield response across the 30 production group C3MP sites for each of the 99 senstivity tests (what we call the simualted mean yields). As noted in Willmott (1984); Legates and McCabe (1999); Snyder et al (2017), NRMS < 1 is one benchmark for adequate model performance, NRMS< 0.5 is a benchmark for good model performance, and NRMS = RMSE = 0 is perfect model 10 performance. We further subdivide these categories and define excellent in-sample performance as NRMS≤ 0.25 for all three response functions; good performance to be 0.25 < N RM S ≤ 0.5 for at least one response function and NRMS≤ 0.25 for at least one response function; adequate performance to be all three response functions having N RM S < 1 but at least one response function with 0.5 < N RM S < 1; and finally poor performance occurs when any one of the three response functions has N RM S ≥ 1.…”

Modeling Land Use and Land Cover Change: Using a Hindcast to Estimate Economic Parameters in gcamland v2.0

“…1 The expected prices and yield would affect farmers' expected rental profits and, thus, land use decisions. (Féménia and Gohin, 2011;Lundberg et al, 2015;Mitra and Boussard, 2012;Roberts and Schlenker, 2013) Linear Linear expectations Linear expectations (Calvin et al, 2017;Snyder et al, 2017) Hybrid Linear…”

“…Bonsch et al (2013) compare simulated land-use change CO2 emissions from MAgPIE to observations, finding that the choice of observation dataset matters for how well the model performs. Calvin et al (2017) and Snyder et al (2017) compare agricultural production and land area simulated by the GCAM model to observations, finding that the model does better for trends than annual values and that some region/crop combinations are better than others. The authors test the use of expectations about yield using a linear forecast as a driver of land use change instead of observed yield, finding that simulations using expected yield better match observations than those using observed yield.…”