Compound Reinforcement Learning: Theory and an Application to Finance

Abstract

Abstract. This paper describes compound reinforcement learning (RL) that is an extended RL based on the compound return. Compound RL maximizes the logarithm of expected double-exponentially discounted compound return in returnbased Markov decision processes (MDPs). The contributions of this paper are (1) Theoretical description of compound RL that is an extended RL framework for maximizing the compound return in a return-based MDP and (2) Experimental results in an illustrative example and an application to finance.

Researches advanced in the application of reinforcement learning

“…The strategy gradient algorithm directly adopts the function approximation method to establish the strategy network, selects the action through the strategy network to get the reward value, and optimizes the parameters of the strategy network along the gradient direction to get the optimal strategy maximum reward value. [8]In the application of the DRL algorithm, the value function algorithm needs to sample actions, so it can only deal with discrete actions, while the strategy gradient algorithm can directly use policy network to search actions, so it can be used to deal with continuous actions. In recent years, the actor-critic structure, which combines value function algorithm with strategy gradient algorithm, has also attracted extensive attention.…”

“…The method of updating the target Q network every N times is adopted to avoid the instability of the target Q network caused by the change of the current Q network during training. [8]The network structure of the DQN algorithm is shown below. 2…”

“…The algorithm adopted is off-policy actor-critic algorithm. [8] The actor-critic algorithm combines the value-based (critic, critic) method with the policy-based (actor, critic) method. At the same time, the learning strategy and value function Actor train strategy according to the value function feedback of critic, while the value function of critic training is updated step by step with time difference method (TD).…”

“…In the subsequent algorithms, deep learning and other methods are introduced to alleviate these problems to a certain extent. [8] Figure 3 Actor-Critic algorithm…”

Reinforcement Learning Using Quantum Boltzmann Machines

“…Reinforcement learning ( [8], known also as neuro-dynamic programming [9]) is an area of optimal control theory at the intersection of approximate dynamic programming and machine learning. It has been used successfully for many applications, in fields such as engineering [10,11], sociology [12,13], and economics [14,15].…”