Evaluating the role judgment plays in forecast accuracy

Abstract

The majority of model-based forecasting efforts today rely on relatively simple techniques of estimation and the subjective adjustment of the model's results to produce forecasts. Published forecasts reflect to a great extent the judgment of the forecaster rather than what the model by itself has to say about the future. This paper examines the role judgment plays in the process of producing a macroeconometric forecast. The debate over the use of adjustment constants to alter the statistical results of a model is outlined and an empirical analysis of forecasts generated by the Michigan Quarterly Econometric Model of the US economy is presented using a unique data set which isolates the role of judgment in the forecasting process. KEY WORDS Judgment Macroeconometric models Adjustment constants Forecast accuracyFor over 50 years economists have been using econometric methods to produce models of economic behavior. Since the early 1950s, with Lawrence Klein's development of the first macroeconometric forecasting model of the US economy, the majority of model-based forecasting efforts have relied on relatively simple techniques of estimation and the subjective, ad hoc adjustment of the model's statistical results to produce forecasts of future events. While economic models have become much larger since that time, and the release of the forecasts more frequent, the standard operating procedures remain relatively unchanged-despite significant advancements in the theory and practice of econometrics and the availability of economic data. Published forecasts of economic behavior still reflect to a great extent the judgment and expertise of the forecaster rather than what the model itself has to say about the future given recent economic events. This is in direct contrast with modeling activities in other disciplines. In his evaluation of economic forecasting methods, Stephen McNees (1988) notes that 'Psychologists have long found that models . . . produce more accurate predictions than the unsupported, global intuition of acknowledged experts'. Meteorology is another example of a science which has made significant strides in the areas of modeling techniques, the use of higher-frequency data, and 'See the essays by Lawrence Klein (1982,1991) and Marjorie Brazer (1982) for a review of the early development of the econometric forecasting industry and the role played by the Economics Department at the University of Michigan.

A non-additive decision-aid for venture capitalists' investment decisions

“…The decision-aid is to assist VCs, not replace them. In fact, there is evidence to suggest that a combination of human judgment and decision models results in more accurate decisions than either method alone (Whitecotton et al, 1998;Blattberg and Hoch, 1990;Donihue, 1993). While the research in this article takes an important step towards highlighting the importance of decision-aids for VCs, more research is required into the development of decisionaids and their use within VC firms.…”

Do ‘big losses’ in judgmental adjustments to statistical forecasts affect experts’ behaviour?

“…Interestingly, they showed that forecasters are more likely to adjust the forecasts that would have produced the largest forecast errors had the statistical forecasts remained unrevised. Other researchers have provided further evidence on the efficacy of judgmentally adjusted forecasts in macroeconomics (Donihue, 1993;McNees, 1990;Turner, 1990), accounting earnings (Brown, 1988) and business forecasting (Vere & Griffiths, 1995;Wolfe & Flores, 1990). Syntetos, Nikolopoulos, and Boylan (2010) showed that in addition to the improvements in performance as measured by traditional error metrics, judgmental adjustments of demand forecasts also result in significant reductions in inventory costs.…”

“…The common factor in these studies is that when important domain knowledge is missing from the statistical forecasts, this can be integrated efficiently into the operational forecasts by applying judgmental adjustments to improve performance. However, two key elements affecting the success of an intervention are the reliability and importance of the missing information (Goodwin & Fildes, 1999) and the requirement that humans should not discount reliable statistical forecasts (Donihue, 1993).…”

The design features of forecasting support systems and their effectiveness

“…Evidence has accumulated that integrating such judgement with quantitative forecasts when the manager has access to information that has not been taken into account by the quantitative method leads to improved accuracy (e.g. see Donihue 1993, Mathews and Diamantopoulos ,1990, Goodwin and Fildes,1999. However, judgmental adjustments are also often applied unnecessarily, when the manager has no extra information to bring to the forecast, with deleterious effects on forecast accuracy O'Connor, 1995, Sanders andRitzman, 2001).…”