Towards robust statistical inference for complex computer models

Abstract

Ecologists increasingly rely on complex computer simulations to forecast ecological systems. To make such forecasts precise, uncertainties in model parameters and structure must be reduced and correctly propagated to model outputs. Naively using standard statistical techniques for this task, however, can lead to bias and underestimation of uncertainties in parameters and predictions. Here, we explain why these problems occur and propose a framework for robust inference with complex computer simulations. After having identified that model error is more consequential in complex computer simulations, due to their more pronounced nonlinearity and interconnectedness, we discuss as possible solutions data rebalancing and adding bias corrections on model outputs or processes during or after the calibration procedure. We illustrate the methods in a case study, using a dynamic vegetation model. We conclude that developing better methods for robust inference of complex computer simulations is vital for generating reliable predictions of ecosystem responses.

Bridging implementation gaps to connect large ecological datasets and complex models

“…In interior Alaska, rapidly improving understanding and quantitative predictions of forest distributions is necessary for ecological management under changing climate. Ecologists can increase the pace of model improvements by continuing to build on a growing suite of existing model calibration tools (Fer et al, 2018 ; Oberpriller et al, 2021 ; Pietzsch et al, 2020 ; Speich et al, 2021 ; Tao et al, 2020 ) and team members by trying new methods in different contexts. We provided the first example of a forest model calibration using tree‐ring data allowing us to demonstrate three pragmatic approaches to proceed with model calibration: connecting model outputs to data‐generating processes, determining data‐driven starting conditions from a suite of model simulations, and reducing a high‐dimensional model calibration problem a priori to model fitting for computational efficiency.…”

“…Combining these techniques to understand mechanisms and improve predictions is a promising path forward (Reichstein et al, 2019 ; Wikle & Hooten, 2010 ). Examples of such ecological applications (Fer et al, 2018 ; Oberpriller et al, 2021 ; Pietzsch et al, 2020 ; Speich et al, 2021 ; Tao et al, 2020 ) are critical for moving beyond implementation barriers and solving urgent ecological problems by bringing these promising tools to larger application‐based audiences.…”

“…A more informal approach involves “tuning” parameters where the user alters parameters until the output more closely matches the observations. More formal techniques for model calibration involve either informing parameters directly with data (LeBauer et al, 2013 ) or iteratively updating parameters or parameter distributions as data become available (Fer et al, 2018 ; Oberpriller et al, 2021 ; Pietzsch et al, 2020 ; Speich et al, 2021 ; Tao et al, 2020 ). Similarly, modelers may choose to calibrate all parameters in a computational model, which we refer to as “full model calibration,” or a modeler can choose to focus on the most sensitive model parameters using an “a priori” parameter sensitivity analysis.…”

“…Only recently have they been proposed as a useful framework for calibration of agent‐based models in ecology (Banks & Hooten, 2021 ; Hooten et al, 2020 ). Furthermore, it has been shown that GPs can account for model structural error and model parameter error simultaneously (Oberpriller et al, 2021 ), offering an advantage to conventional model calibration techniques (Hartig et al, 2011 ). As GPs become increasingly popular, many statistical software packages built for the non‐specialist are being developed (Erickson et al, 2018 ; Fer et al, 2018 ; Liu et al, 2020 ).…”

Climate and parameter sensitivity and induced uncertainties in carbon stock projections for European forests (using LPJ-GUESS 4.0)

“…Among the reasons for such different results is the inherent uncertainty in climate scenarios (Saraiva et al, 2019), model structural uncertainty (Bugmann et al, 2019;Oberpriller et al, 2021;Prestele et al, 2016) as well as uncertainty about the model parametrization (Grimm, 2005), which in turn make models' projections themselves uncertain (Dietze, 2017). When considering the impact of these uncertainties for directing research (Tomlin, 2013), but also to interpret and understand projections (Dietze et al, 2018), it is of immense value to know which factors drive these uncertainties.…”