Reinforcement learning for optimized trade execution

Abstract

We present the first large-scale empirical application of reinforcement learning to the important problem of optimized trade execution in modern financial markets. Our experiments are based on 1.5 years of millisecond time-scale limit order data from NASDAQ, and demonstrate the promise of reinforcement learning methods to market microstructure problems. Our learning algorithm introduces and exploits a natural "low-impact" factorization of the state space.

Strategic execution in the presence of an uninformed arbitrageur

“…Developing tractable models that account for such data remains a challenge. One initiative to incorporate limit order book data into the decision process is presented by Nevmyvaka et al (2006).…”

Learning unfair trading: A market manipulation analysis from the reinforcement learning perspective

“…A model that fits the problem of manipulation under the representation Fig. 1 is that of a Markov Decision Process (MDP) [18], [22]. In general, an MDP is defined by the tuple {S, A, T, R}, where S and A are sets of states and actions, respectively (s ∈ S and a ∈ A), R is the set of rewards (r ∈ R), and T is a set of transition probabilities ({P (s |s, a)} ∈ T , where P (s |s, a) represents the probability of transitioning to state s from s after action a).…”

Evolutionary Computation and Trade Execution

“…that can for example vary their aggressiveness, are superior to static strategies. Nevmyvaka [30] proposed dynamic price adjustment strategy, where limit order's price is revised every 30 seconds adapting to the changing market state. Wang [35] proposed a dynamic focus strategy, which dynamically adjusts volume according to real-time update of state variables such as inventory and order book imbalance, and showed that dynamic focus strategy can outperforms a static limit order strategy.…”

“…The objective function we used here is ratio of the difference between the VWAPs of the 30 orders and the entire executed orders generated from the ASM simulation to the entire executed orders' VWAP, which are V W AP 30 and V W AP global respectively. For both buy and sell orders, the smaller the VWAP Ratio, the better the strategy is.…”