Learning to trade via direct reinforcement

Abstract

We present methods for optimizing portfolios, asset allocations, and trading systems based on direct reinforcement (DR). In this approach, investment decision-making is viewed as a stochastic control problem, and strategies are discovered directly. We present an adaptive algorithm called recurrent reinforcement learning (RRL) for discovering investment policies. The need to build forecasting models is eliminated, and better trading performance is obtained. The direct reinforcement approach differs from dynamic programming and reinforcement algorithms such as TD-learning and Q-learning, which attempt to estimate a value function for the control problem. We find that the RRL direct reinforcement framework enables a simpler problem representation, avoids Bellman's curse of dimensionality and offers compelling advantages in efficiency. We demonstrate how direct reinforcement can be used to optimize risk-adjusted investment returns (including the differential Sharpe ratio), while accounting for the effects of transaction costs. In extensive simulation work using real financial data, we find that our approach based on RRL produces better trading strategies than systems utilizing Q-learning (a value function method). Real-world applications include an intra-daily currency trader and a monthly asset allocation system for the S&P 500 Stock Index and T-Bills.

A Multiagent Approach to $Q$-Learning for Daily Stock Trading

“…In [15], an adaptive algorithm, which was named recurrent reinforcement learning, for direct reinforcement was proposed, and it was used to learn an investment strategy online. Later, Moody and Saffell [16] have shown how to train trading systems via direct reinforcement. Performance of the learning algorithm proposed in [16] was demonstrated through the intraday currency trader and monthly asset allocation system for S&P 500 stock index and T-Bills.…”

“…Later, Moody and Saffell [16] have shown how to train trading systems via direct reinforcement. Performance of the learning algorithm proposed in [16] was demonstrated through the intraday currency trader and monthly asset allocation system for S&P 500 stock index and T-Bills.…”

Comparing Policy Gradient and Value Function Based Reinforcement Learning Methods in Simulated Electrical Power Trade

“…Using some measure of risk adjusted return (as in [8]) might be of interest in the context of simulated electricity trade and this would simply involve the definition of a new task and would not require any modification of the environment.…”

“…In [8] and [9] a recurrent gradient method was used to optimise financial investment performance without price forecasting and in [38] a modified version of REINFORCE is used to simulate a marketplace for grid computing resources.…”

“…They can be used with function approximation techniques without suffering from the problems that mar value function based methods. They have been successfully applied in several types of operational setting, including robotic control [7], financial trading [8], [9] and network routing [10], but they have not been previously applied in simulations of competitive electricity trade.…”