Testing out-of-sample portfolio performance

Abstract

This paper studies the quality of portfolio performance tests based on out-of-sample returns. By disentangling the components of out-of-sample performance we show that observed differences are driven to a large extent by the differences in estimation risk. Our Monte Carlo study reveals that the puzzling empirical findings of inferior performance of theoretically superior strategies mainly result from the low power of these tests. Thus our results provide an explanation why the null hypothesis of equal performance of the simple equally weighted portfolio compared to many alternatives, theoretically superior strategies cannot be rejected in many out-of-sample horse races. Our findings turn out to be robust with respect to different designs and the implementation strategies of the tests. For the applied researcher we provide some guidance to cope with the problem of low power. In particular, we show by the means of a novel pretest-based portfolio strategy, how the information of performance tests can be used optimally.

Horses for courses: Mean-variance for asset allocation and 1/N for stock selection

“…This is consistent with the common finding that tests of 1/N versus MV in a one stage process are often not as clear cut as our two stage results. Kazak and Pohlmeier (2019) report that statistical tests of portfolio models for out-performance have low statistical power because such tests are heavily influenced by the length of the out-of-sample period and estimation noise.…”

Nonparametric performance hypothesis testing with the information ratio

“…On the other hand, a recent study of Kazak & Pohlmeier (2019) performed one-sided and twosided tests on out-of-sample performance of portfolios constructed naïvely and through global minimum variance in terms of their Sharpe ratio and certainty equivalent. The methods used were delta and bootstrap methods for which a multivariate normal distribution is assumed to be the distribution of portfolio returns.…”

Sticky Stock Market Analysts

“…However, many of these approaches have yet to prove their suitability for actual ex-ante forecasts. Their informative value for ex-ante forecasts might be limited due to, for example, differences in estimation risk and low statistical power (Kazak and Pohlmeier 2019).…”

“…New forecasting methods are constantly being discussed: econometric models (Goyal et al 2021;Chen and Vincent 2016;Welch and Goyal 2008), artificial neural networks (Rajab and Sharma 2019;Atsalakis and Valavanis 2009), artificial intelligence (Mallikarjuna and Rao 2019), capital market simulations with multi-agent models (Yang et al 2020;Krichene and El-Aroui 2018;Arthur et al 1997), modelling based on the expectations of capital market agents (Atmaz et al 2021;Greenwood and Shleifer 2014), and neuro-psycho-economics approaches (Ortiz-Teran et al 2019;Kandasamy et al 2016;Werner et al 2009). However, testing these approaches using ex-post forecasts in an out-ofsample data domain repeatedly leads to apparent forecasting successes that then may not materialize in real ex-ante settings (Kazak and Pohlmeier 2019). When the variability of reality is systematically underestimated, this can contribute towards very costly errors in the field of stock market forecasts.…”