Performance functions and reinforcement learning for trading systems and portfolios

Abstract

We propose to train trading systems and portfolios by optimizing objective functions that directly measure trading and investment performance. Rather than basing a trading system on forecasts or training via a supervised learning algorithm using labelled trading data, we train our systems using recurrent reinforcement learning (RRL) algorithms. The performance functions that we consider for reinforcement learning are profit or wealth, economic utility, the Sharpe ratio and our proposed differential Sharpe ratio. The trading and portfolio management systems require prior decisions as input in order to properly take into account the effects of transactions costs, market impact, and taxes. This temporal dependence on system state requires the use of reinforcement versions of standard recurrent learning algorithms. We present empirical results in controlled experiments that demonstrate the efficacy of some of our methods for optimizing trading systems and portfolios. For a long/short trader, we find that maximizing the differential Sharpe ratio yields more consistent results than maximizing profits, and that both methods outperform a trading system based on forecasts that minimize MSE. We find that portfolio traders trained to maximize the differential Sharpe ratio achieve better risk‐adjusted returns than those trained to maximize profit. Finally, we provide simulation results for an S&P 500/TBill asset allocation system that demonstrate the presence of out‐of‐sample predictability in the monthly S&P 500 stock index for the 25 year period 1970 through 1994. Copyright © 1998 John Wiley & Sons, Ltd.

A Multiagent Approach to $Q$-Learning for Daily Stock Trading

“…While the reinforcement learning is promising, introduction of these considerations will make the problem more complex. Therefore, one of the future AND 15 research problems will be to make the reinforcement learning formulation with these considerations tractable.…”

“…Another portfolio management system built by use of Q-learning was presented in [14] where absolute profit and relative risk-adjusted profit were considered as performance functions to train a system. In [15], an adaptive algorithm, which was named recurrent reinforcement learning, for direct reinforcement was proposed, and it was used to learn an investment strategy online. Later, Moody and Saffell [16] have shown how to train trading systems via direct reinforcement.…”

PAMR: Passive aggressive mean reversion strategy for portfolio selection

“…For some process-dependent investors (Moody et al 1998), it is important to evaluate risk and risk-adjusted return of portfolios (Sharpe 1963(Sharpe , 1994. One common way to achieve this is to use annualized standard deviation of daily returns to measure the volatility risk and annualized Sharpe Ratio (SR) to evaluate the risk-adjusted return.…”

Comparing Policy Gradient and Value Function Based Reinforcement Learning Methods in Simulated Electrical Power Trade

“…In [8] and [9] a recurrent gradient method was used to optimise financial investment performance without price forecasting and in [38] a modified version of REINFORCE is used to simulate a marketplace for grid computing resources.…”

“…They can be used with function approximation techniques without suffering from the problems that mar value function based methods. They have been successfully applied in several types of operational setting, including robotic control [7], financial trading [8], [9] and network routing [10], but they have not been previously applied in simulations of competitive electricity trade.…”