Action-specialized expert ensemble trading system with extended discrete action space using deep reinforcement learning

Abstract

Despite active research on trading systems based on reinforcement learning, the development and performance of research methods require improvements. This study proposes a new action-specialized expert ensemble method consisting of action-specialized expert models designed specifically for each reinforcement learning action: buy, hold, and sell. Models are constructed by examining and defining different reward values that correlate with each action under specific conditions, and investment behavior is reflected with each expert model. To verify the performance of this technique, profits of the proposed system are compared to those of single trading and common ensemble systems. To verify robustness and account for the extension of discrete action space, we compared and analyzed changes in profits of the three actions to our model's results. Furthermore, we checked for sensitivity with three different reward functions: profit, Sharpe ratio, and Sortino ratio. All experiments were conducted with S&P500, Hang Seng Index, and Eurostoxx50 data. The model was 39.1% and 21.6% more efficient than single and common ensemble models, respectively. Considering the extended discrete action space, the 3-action space was extended to 11-and 21-action spaces, and the cumulative returns increased by 427.2% and 856.7%, respectively. Results on reward functions indicated that our models are well trained; results of the Sharpe and Sortino ratios were better than the implementation of profit only, as in the single-model cases. The Sortino ratio was slightly better than the Sharpe ratio.

Deep Reinforcement Learning For Trading - A Critical Survey

“…The action space is generally the simple discrete {buy, hold, sell} but there are some which use more actions to denote a different number of shares or quantity to buy or sell, as in [68]. If dealing with multiple assets either a continuous multi-dimensional action space is used or the problem is split into selecting the number of shares / quantity (for example with a deep learning module [69,70]) and separately selecting the type of order (buy, sell).…”

Deep Reinforcement Learning for Trading—A Critical Survey

“…The action space is generally the simple discrete {buy, hold, sell}, but there are some which use more actions to denote a different number of shares or quantity to buy or sell, as in [74]. If dealing with multiple assets, either a continuous multi-dimensional action space is used, or the problem is split into selecting the number of shares/quantity (for example, with a deep learning module [75,76]) and separately selecting the type of order (buy, sell).…”

A Novel Evaluation Strategy to Artificial Neural Network Model Based on Bionics

“…In essence, it is a simple mapping of the biological nervous system. With the in-depth study of the ANN, researchers have made breakthroughs in such fields as pattern recognition [ 2 ], natural language processing [ 3 ], intelligent control [ 4 , 5 ], expert system [ 6 ] and predictive analysis [ 7 , 8 ], and the network structure and function is developing towards complexity and intelligence. The development of the ANN has been distributed over three stages [ 9 ]: the first-generation neural network is a very simple model, which is composed of threshold neurons, such as multi-layer perceptron [ 10 ] and Hopfield neural network [ 11 ].…”