Evaluating forecast performance with state dependence

Abstract

We propose a novel forecast evaluation methodology to assess models' absolute and relative forecasting performance when it is a state-dependent function of economic variables. In our framework, the forecasting performance, measured by a forecast error loss function, is modeled via a hard or smooth threshold model with unknown threshold values. Existing tests either assume a constant out-of-sample forecast performance or use non-parametric techniques robust to time-variation; consequently, they may lack power against state-dependent predictability. Our tests can be applied to relative forecast comparisons, forecast encompassing, forecast efficiency, and, more generally, moment-based tests of forecast evaluation. Monte Carlo results suggest that our proposed tests perform well in finite samples and have better power than existing tests in selecting the best forecast or assessing its efficiency in the presence of state dependence. Our tests uncover "pockets of predictability" in U.S. equity premia; although the term spread is not a useful predictor on average over the sample, it forecasts significantly better than the benchmark forecast when real GDP growth is low. In addition, we find that leading indicators, such as measures of vacancy postings and new orders for durable goods, improve the forecasts of U.S. industrial production when financial conditions are tight.

The Bias of the ECB Inflation Projections: a State-Dependent Analysis

“…First, we outline how to test for bias in a linear framework. Then, we present the novel forecast evaluation testing methodology proposed by Odendahl et al (2023) and we show how it can be applied to testing for bias in a state-dependent setting. Finally, we describe the simplified approach to test for state-dependent bias used in previous studies.…”

“…Specifically, we ask whether the ECB makes systematic forecast errors depending on the level of inflation observed when making its projections. To answer this question, we apply the newly developed state-dependent bias test by Odendahl et al (2023). The test is based on a regression model where the forecast errors are allowed to depend on both a linear (an intercept, in our case) and a non-linear term.…”

“…We construct a real time dataset and produce inflation forecasts for the same sample for which the ECB projections are available. We then apply the Odendahl et al (2023) bias test to these real time forecasts. For all the models we find evidence of state-dependent bias, but of the opposite sign than the one observed for the ECB projections: the forecasting models tend to overpredict when inflation is higher than the estimated threshold, indicating that inflation is predicted to revert to its long term mean too slowly.…”

“…Therefore, we relate the bias to the ECB's mandate and its forecasting process. Moreover, this is the first paper that applies the methodology by Odendahl et al (2023) to testing for state-dependent bias with an unknown threshold value that determines the switch between states. We depart from these earlier studies also because we analyze the ECB projections, which have received little attention so far.…”

Forecasting Exchange Rates Comparison In The Presence Of Instabilities

“…As stated by Rossi (2021), "instabilities are widespread in economic time series". Odendahl et al,( 2022) also stressed that "despite various predictive comparison tests that have been proposed in the literature to help predictors in model selection, no single model is usually the best overall. Predictive performance is typically prone to volatility and is sample dependent".…”