Multi-Horizon Forecast Comparison

Abstract

We introduce tests for multi-horizon superior predictive ability (SPA). Rather than comparing forecasts of different models at multiple horizons individually, we propose to jointly consider all horizons of a forecast path. We define the concepts of uniform and average SPA. The former entails superior performance at each individual horizon, while the latter allows inferior performance at some horizons to be compensated by others. The article illustrates how the tests lead to more coherent conclusions, and how they are better able to differentiate between models than the single-horizon tests. We provide an extension of the previously introduced model confidence set to allow for multi-horizon comparison of more than two models. Simulations demonstrate appropriate size and high power. An illustration of the tests on a large set of macroeconomic variables demonstrates the empirical benefits of multi-horizon comparison.

Bayesian VAR Forecasts, Survey Information and Structural Change in the Euro Area

“…Tables A.1 and A.2 show results for tilting different variables. Positive (bold) numbers indicate superior predictive ability of the tilted BVAR relative to the raw BVAR and * , * * , * * * denote statistical significance resulting from the bootstrap procedure of Quaedvlieg (2019). The multi-horizon test confirms the findings in Tables 1, 3, C.3 and C.5.…”

“…Finally, tilting all three variable jointly in Panel E shows sizeable improvements for all four variables. The appendix additionally shows results of the multi-horizon forecast comparison test of Quaedvlieg (2019), which controls the family-wise error rate of the multiple hypothesis tests displayed in Table 1 and the tables thereafter.…”

“…of the multi-horizon forecast comparison test of Quaedvlieg (2019), which tests for superior predictive ability of the quarter-on-quarter forecasts while controlling the family-wise error rate of the multiple hypothesis tests. We conduct the test for horizon 1 to 4 and 1 to 8 (note that this is different from the year-on-year tests of h = 4 * and h = 8 * .…”

Testing for Differences in Path Forecast Accuracy: Forecast-Error Dynamics Matter

“…Then, depending on howσ N is treated, it is possible to recover the trace test statistic as proposed by Capistrán (2006) or an unweighted version of the average superior predictive ability (aSPA) test discussed in Quaedvlieg (2017). Although it is less obvious, (3.10) also nests differences based on Doan et al (1984)'s log determinant metric across multiple horizons.…”

“…the single horizon version of (3.8)), (3) the unweighted general likelihood ratio test statistic from equation (3.8), and (4) the GFESM test of Corollary 2 from (4.5). The first test represents existing approaches in the literature as seen in Capistrán (2006) and Quaedvlieg (2017) that do not explicitly target differences in covariances or dynamics. The second test is effectively the log determinant measure from Doan et al (1984) applied to the test described in Quaedvlieg (2017) which accounts for covariances but not dynamics.…”