Proposal and extensive test of a calibration protocol for crop phenology models

Abstract

A major effect of environment on crops is through crop phenology, and therefore, the capacity to predict phenology as a function of soil, weather, and management is important. Mechanistic crop models are a major tool for such predictions. It has been shown that there is a large variability between predictions by different modeling groups for the same inputs, and therefore, a need for shared improvement of crop models. Two pathways to improvement are through improved understanding of the mechanisms of the modeled system, and through improved model parameterization. This article focuses on improving crop model parameters through improved calibration, specifically for prediction of crop phenology. A detailed calibration protocol is proposed, which covers all the steps in the calibration work-flow, namely choice of default parameter values, choice of objective function, choice of parameters to estimate from the data, calculation of optimal parameter values and diagnostics. For those aspects where knowledge of the model and target environments is required, the protocol gives detailed guidelines rather than strict instructions. The protocol includes documentation tables, to make the calibration process more transparent. The protocol was applied by 19 modeling groups to three data sets for wheat phenology. All groups first calibrated their model using their "usual" calibration approach. Evaluation was based on data from sites and years not represented in the training data. Compared to usual calibration, l calibration following the new protocol significantly reduced the error in predictions for the evaluation data, and reduced the variability between modeling groups by 22%.

Why is there so much variability in crop multi-model studies?

“…This study revisits three simulation experiments on wheat phenology that have been previously reported (Wallach et al, 2021a(Wallach et al, , 2021b(Wallach et al, , 2022. In the previous studies, the emphasis was on prediction accuracy, whereas in the present study the simulation results are reanalyzed with the objective of estimating the contribution of parameter uncertainty to the overall uncertainty of the phenology prediction.…”

“…Our second estimate of parameter uncertainty is based on a comparison, for multiple modeling groups, between two different calibration procedures, their usual procedure, and a new protocol procedure. In general, there were major differences between the two calibration procedures for each group (Wallach et al, 2022). Both are plausible in the sense that they were developed and used by modeling teams well-versed in the use of their model structure.…”

“…There is no consensus on the best calibration approach for crop models. However, there have been recent attempts to define a calibration protocol that could be applied generally to phenology models, and that has been found both to reduce uncertainty and to reduce mean squared error of prediction (Wallach et al, 2022).…”

“…The proposed calibration protocol is described in detail in Wallach et al (2022). The protocol covers all steps in phenology model calibration and includes templates for documenting the different steps.…”

“…How to sample from plausible parameter vectors on the other hand is not clear. Wallach et al (2022) identified several steps in crop model calibration, including the choice of default parameter values, the choice of observed data to use in the objective function for frequentist calibration, the choice of parameters to be estimated from the data and finally the choice of algorithm and software for doing the calibration. They showed that different modeling groups made different choices, and that even different modeling groups using the same model structure made different choices.…”