A Note on Qualitative Forecast Evaluation

Abstract

Traditionally, the 2 x 2 contingency table method has been used for qualitative evaluation of forecasts. However, the conclusions drawn based on this method could be misleading because it does not account for the direction of the turning or no turning points. A 4 x 4 contingency table which overcomes this weakness and which gives more information on the qualitative performance of the forecast is suggested.Several qualitative and quantitative measures have been used in the literature to evaluate the performance of alternative forecasting techniques. Though the usefulness of these measures depends on the purpose for which the forecast is made, both qualitative and quantitative measures can be important in guiding economic agents in making decisions. The most commonly used measures of quantitative evaluation of forecasts are mean forecast error, mean absolute error, mean squared error, root mean squared error, percentage root mean squared error and goodness of fit. As far as qualitative evaluation is concerned, researchers have used directional change, binomial probability functions, and the turning point method. It is the purpose of this note to demonstrate a limitation to the now popular turning point method of qualitative evaluation, which potentially can lead researchers to incorrect interpretation of their results. We will then offer an alternative qualitative measure which can provide more insight into the forecast evaluation. Qualitative MeasuresSeveral researchers (Hee; Kulshreshtha, Spriggs, and Akinfemiwa; Leuthold; Meyers, Havlicek, and Henderson) have used directional change as a measure of qualitative forecast evaluation. Usually, this is a matter of Gopal Naik and Raymond M. Leuthold are, respectively, a graduate student and a professor, Department of Agricultural Economics, University of Illinois.This study was a part of Hatch Project 05-381 of the University of Illinois Agricultural Experiment Station.Review was coordinated by Rulon Pope, associate editor.simply noting whether the forecasted variable is higher or lower than the current actual variate and if the subsequent move of the actual variable corresponded. The number or percentage of correct forecast directions is usually noted. Hee and Meyers, Havlicek, and Henderson used a binomial probability function in order to compare the number of correct directional changes the forecast has predicted with the number of correct directional changes that could be obtained purely by a chance factor. Kulshreshtha, Spriggs, and Akinfemiwa used the ratio of correct directional moves forecasted to total directional moves over the entire forecast period. The ratio is compared with a random correct directional move of .5. Theil suggested using the number of over or under predictions and the turning point method to evaluate the qualitative performance. This turning point method is now a popular qualitative measure frequently used by researchers Brandt 1979, 1981;Bourke;Bessler 1981, 1984;Harris and Leuthold;Hudson and Capps;Kost;Kulshreshtha and Rosaasen). This m...

Nowcasting and Forecasting the Monthly Food Stamps Data in the US Using Online Search Data

“…A special case of directional accuracy is the ability to predict a turning point, which is a change in the direction of movement of the variable under investigation, and it exists if (peak turning point) or (trough turning point), see [74] and [75] for details.…”

Directional forecasts of GDP and inflation: a joint evaluation with an application to Federal Reserve predictions

“…The justification for jointly evaluating the Fed's GDP growth and inflation forecasts is that the Taylor (1993) Naik and Leuthold (1986) also used a 4x4 contingency table in their qualitative analysis of forecasting performance. Their study focused on a different topic-the ability to predict turning points.…”

A Comparative Analysis of Time Series Approaches to Modeling Intersectoral and Intercounty Employmet Linkages in Rural Regional Labor Markets*

“…We judged model performance on the basis of the magnitude of mean squared errors from the twelve-step ahead forecast of the changes in monthly employment in each sector, an asymptotic test on predictive accuracy (Diebold and Mariano, 19911, and a test on the reliability of turning point predictions (Naik and Leuthold, 1986).…”

“…In addition to models with small forecast errors, policymakers are interested in models that accurately predict the direction of changes in employment. We evaluated the accuracy with which each of the five models predicted turning points in the out-of-sample observations, using the 4 x 4 contingency table approach developed in Naik and Leuthold (1986). Our results are summarized in Table 6.…”