Model evaluation for policy insights: Reflections on the forum process

Abstract

Model evaluation is best considered as a process for communicating with the policymaking or policyadvising community. Six decades of energy modelling have witnessed increasing complexity in these systems, a situation that raises a number of important challenges in using them effectively in policymaking organizations. When used as a learning rather than forecasting tool, these systems can be evaluated individually one by one or through joint efforts to compare them in multi-model exercises. After summarizing the evolution of energy modelling and efforts to evaluate them since the first oil embargo, this essay provides a guide to future evaluation collaborations by highlighting a few challenges that would improve the value of these studies for the policymaking community. These challenges range broadly and cover topics such as enhancing the engagement of the model user, ventilating the models' complexity with intuitive insights, using simple models to demonstrate key parameters or responses, applying judicious occasional meta-analysis when there is value added, reporting model responses and calibrating them for decisionmakers, considering retrospective evaluation for a past period (when possible), selecting standardized or modeler-choice baseline conditions, selectively developing policy or diagnostic alternative cases, and institutionalising the model evaluation process for a specific topic or region.