OptIC project: An intercomparison of optimization techniques for parameter estimation in terrestrial biogeochemical models

Abstract

[1] We describe results of a project known as OptIC (Optimisation InterComparison) for comparison of parameter estimation methods in terrestrial biogeochemical models. A highly simplified test model was used to generate pseudo-data to which noise with different characteristics was added. Participants in the OptIC project were asked to estimate the model parameters used to generate this data, and to predict model variables into the future. Ten participants contributed results using one of the following methods: Levenberg-Marquardt, adjoint, Kalman filter, Markov chain Monte Carlo and genetic algorithm. Methods differed in how they locate the minimum (gradient-descent or global search), how observations are processed (all at once sequentially), or the number of iterations used, or assumptions about the statistics (some methods assume Gaussian probability density functions; others do not). We found the different methods equally successful at estimating the parameters in our application. The biggest variation in parameter estimates arose from the choice of cost function, not the choice of optimization method. Relatively poor results were obtained when the model-data mismatch in the cost function included weights that were instantaneously dependent on noisy observations. This was the case even when the magnitude of residuals varied with the magnitude of observations. Missing data caused estimates to be more scattered, and the uncertainty of predictions increased correspondingly. All methods gave biased results when the noise was temporally correlated or non-Gaussian, or when incorrect model forcing was used. Our results highlight the need for care in choosing the error model in any optimization.

Leveraging 35 years of <i>Pinus taeda</i> research in the southeastern US to constrain forest carbon cycle predictions: regional data assimilation using ecosystem experiments

“…Our approach allowed the estimation of the probability distribution of forest biomass before uncertainty is added through measurement. Considering that the method of DA can potentially have a large influence on posterior parameter distributions (Trudinger et al, 2007), future research should focus on comparing the hierarchical approach presented here to other approaches by using the same data constraints with alternative cost functions.…”

“…We numerically estimated the joint posterior distribution using the Monte Carlo Markov Chain-Metropolis Hasting (MCMC-MH) algorithm (Zobitz et al, 2011). This approach has been widely used to approximate parameter distributions in ecosystem DA research (Fox et al, 2009;Trudinger et al, 2007;Williams et al, 2005;Zobitz et al, 2011). Briefly, the algorithm proposed new values for the model parameters, uncertainty parameters, latent states, and FR.…”

Linking big models to big data: efficient ecosystem model calibration through Bayesian model emulation

“…As this dataset was generated by the model itself, this approach allows us to assume that we have the perfect model (Trudinger et al, 2007;Fox et al, 2009). We compared the emulator run in three rounds to an emulator fit to the same number of knots in a single run to test whether increasing the number of knots iteratively is more effective than proposing the same number of knots in the beginning all-at-once.…”

Mechanistic scaling of ecosystem function and dynamics in space and time: Ecosystem Demography model version 2

“…Accordingly, there have been a number of recent efforts that have used ecosystem measurements to constrain the behavior of terrestrial biosphere models [Raupach et al, 2005;Trudinger et al, 2007]. Such techniques can be used to optimize model parameters and obtain information on parameter uncertainties and covariances.…”