Deep Reinforcement Learning in Agent Based Financial Market Simulation

Abstract

Prediction of financial market data with deep learning models has achieved some level of recent success. However, historical financial data suffer from an unknowable state space, limited observations, and the inability to model the impact of your own actions on the market can often be prohibitive when trying to find investment strategies using deep reinforcement learning. One way to overcome these limitations is to augment real market data with agent based artificial market simulation. Artificial market simulations designed to reproduce realistic market features may be used to create unobserved market states, to model the impact of your own investment actions on the market itself, and train models with as much data as necessary. In this study we propose a framework for training deep reinforcement learning models in agent based artificial price-order-book simulations that yield non-trivial policies under diverse conditions with market impact. Our simulations confirm that the proposed deep reinforcement learning model with unique task-specific reward function was able to learn a robust investment strategy with an attractive risk-return profile.

Deep learning in the stock market—a systematic survey of practice, backtesting, and applications

“…The authors in Maeda et al (2020) propose a market DRL framework to help improve the performance of DL algorithms using a combination of DRL and LSTM with simulated market data. By simulating the order books for limit, market, and cancel orders, they are able to maximize returns.…”

Policy Gradient Stock Gan for Realistic Discrete Order Data Generation in Financial Markets

“…In the artificial market simulation context, Maeda et al [24] attempted to make a model learning the better trading strategy via augmented data by artificial market. Returning to the basics of artificial market simulation, Edmonds et al [9] argued that agent-based simulation is useful for social sciences.…”

Deep Hawkes Process for High-Frequency Market Making

“…The market simulator is an essential tool for designing, evaluating and backtesting algorithmic trading strategies under various market scenarios. The market is flooded with various proprietary and open-source financial simulation frameworks, but the constraints associated with licences, application interfaces and software design limit their usability (Maeda et al, 2020;Izumi and Toriumi, 2009). In this paper, we have designed a multi-asset market simulator from scratch, which is scalable to markets of substantial size.…”

“…The exemplary predictive performance of deep-learning models has encouraged researchers to augment order book data with agent-based artificial market simulation, for the purpose of investigating algorithmic trading strategies (Maeda et al, 2020). The success of the model is dependent on the simulation framework of the financial market being close to realism.…”