Action Branching Architectures for Deep Reinforcement Learning

Abstract

Discrete-action algorithms have been central to numerous recent successes of deep reinforcement learning. However, applying these algorithms to high-dimensional action tasks requires tackling the combinatorial increase of the number of possible actions with the number of action dimensions. This problem is further exacerbated for continuous-action tasks that require fine control of actions via discretization. In this paper, we propose a novel neural architecture featuring a shared decision module followed by several network branches, one for each action dimension. This approach achieves a linear increase of the number of network outputs with the number of degrees of freedom by allowing a level of independence for each individual action dimension. To illustrate the approach, we present a novel agent, called Branching Dueling Q-Network (BDQ), as a branching variant of the Dueling Double Deep Q-Network (Dueling DDQN). We evaluate the performance of our agent on a set of challenging continuous control tasks. The empirical results show that the proposed agent scales gracefully to environments with increasing action dimensionality and indicate the significance of the shared decision module in coordination of the distributed action branches. Furthermore, we show that the proposed agent performs competitively against a state-of-theart continuous control algorithm, Deep Deterministic Policy Gradient (DDPG).

Proactive Content Caching Based on Actor–Critic Reinforcement Learning for Mobile Edge Networks

“…In a branching neural network for the policy function estimation, action space can be separated into several dimensions and each neural network branch estimates a policy function on an action dimension. In [33], A. Tavakoli et al have proven the effectiveness of the branching neural network model.…”

PROMPT: Learning Dynamic Resource Allocation Policies for Edge-Network Applications

“…The resource controller uses a deep Q-network with an action-branching architecture [14] as depicted in Figure 3(b). This architecture features a shared representation layer, followed by distinct action branches, one for each resource control knob.…”

Dynamic Multichannel Access via Multi-agent Reinforcement Learning: Throughput and Fairness Guarantees

“…However, it is still not straightforward how to properly define the advantage of an executed action since the size of the action space A[i] is time-varying for each decision time i. To handle this technical issue, we adapt a branching dueling Q-network architecture [36] and measure the advantage of an executed action separately by each time slot. In particular, we consider action branches [1 : K max ], i.e., time slots [1 : K max ].…”