On measuring quality of a probabilistic commodity forecast for a system that incorporates seasonal climate forecasts

Abstract

Regional commodity forecasts are being used increasingly in agricultural industries to enhance their risk management and decision-making processes. These commodity forecasts are probabilistic in nature and are often integrated with a seasonal climate forecast system. The climate forecast system is based on a subset of analogue years drawn from the full climatological distribution. In this study we sought to measure forecast quality for such an integrated system. We investigated the quality of a commodity (i.e. wheat and sugar) forecast based on a subset of analogue years in relation to a standard reference forecast based on the full climatological set. We derived three key dimensions of forecast quality for such probabilistic forecasts: reliability, distribution shift, and change in dispersion. A measure of reliability was required to ensure no bias in the forecast distribution. This was assessed via the slope of the reliability plot, which was derived from examination of probability levels of forecasts and associated frequencies of realizations. The other two dimensions related to changes in features of the forecast distribution relative to the reference distribution. The relationship of 13 published accuracy/skill measures to these dimensions of forecast quality was assessed using principal component analysis in case studies of commodity forecasting using seasonal climate forecasting for the wheat and sugar industries in Australia. There were two orthogonal dimensions of forecast quality: one associated with distribution shift relative to the reference distribution and the other associated with relative distribution dispersion. Although the conventional quality measures aligned with these dimensions, none measured both adequately. We conclude that a multi-dimensional approach to assessment of forecast quality is required and that simple measures of reliability, distribution shift, and change in dispersion provide a means for such assessment. The analysis presented was also relevant to measuring quality of probabilistic seasonal climate forecasting systems. The importance of retaining a focus on the probabilistic nature of the forecast and avoiding simplifying, but erroneous, distortions was discussed in relation to applying this new forecast quality assessment paradigm to seasonal climate forecasts.

Rainfall and streamflow response to El Niño Southern Oscillation: a case study in a semiarid catchment, Australia

“…In fact, many rainfall gauge networks poorly represent any sort of spatial average for catchments and the Border Rivers catchment is no exception. Further study is needed to explore the skill in forecasts of rainfall and streamflow for different periods of lead times using different climate indicators and multi-dimensional skill determining techniques as reported by Potgieter et al (2003).…”

A forecast quality assessment of an end-to-end probabilistic multi-model seasonal forecast system using a malaria model

“…The literature is limited on probabilistic application forecasts that apply some form of seasonal scale forecasts (Franz et al, 2003;Potgieter et al, 2003) with few reports of probabilistic application models embedded within an ensemble prediction system (e.g. Cantelaube and Terres, 2005;Marletto et al, 2005).…”

Forecasting rainfall based on the Southern Oscillation Index phases at longer lead-times in Australia

“…change in shift and/ or change in dispersion of forecast), and the skill of the forecast (i.e. changes in accuracy relative to a benchmark forecast system) (Murphy 1993;Potgieter et al 2003). The skill of a forecast may be verified in either hindcast or forecast mode.…”