Learning Action Representations for Reinforcement Learning

Chandak, Yash; Theocharous, Georgios; Kostas, James; Jordan, Scott; Thomas, Philip S.

doi:10.48550/arxiv.1902.00183

Cited by 10 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent works have aimed to express the similarities between actions to learn policies more quickly, especially over large action spaces. For example, one approach is to learn action embeddings, which could then be used to learn a policy (Chandak et al, 2019;Chen et al, 2019). Another approach is to directly learn about irrelevant actions and then eliminate them from being selected (Zahavy et al, 2018).…”

Section: Action Reductionmentioning

confidence: 99%

Estimating Q(s,s') with Deep Deterministic Dynamics Gradients

Edwards¹,

Sahni²,

Liu³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper, we introduce a novel form of value function, Q(s, s ), that expresses the utility of transitioning from a state s to a neighboring state s and then acting optimally thereafter. In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that maximize this value. This formulation decouples actions from values while still learning off-policy. We highlight the benefits of this approach in terms of value function transfer, learning within redundant action spaces, and learning off-policy from state observations generated by sub-optimal or completely random policies. Code and videos are available at sites.google.com/view/qss-paper.

show abstract

Section: Action Reductionmentioning

confidence: 99%

Estimating Q(s,s') with Deep Deterministic Dynamics Gradients

Edwards¹,

Sahni²,

Liu³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The method proposed in [22] offers a simple yet effective way to obtain a sparse DNN representation of the training data to assist the DRL agent in better understanding useful and pertinent dynamics in RL tasks. On the other hand, the works in [23,24] investigate the embeddings of action space from theoretical and practical perspectives. Moreover, the Value Prediction Network (VPN) [25] avoids the challenging task of modeling the full environment by only focusing on predicting value/reward of future states.…”

Section: Related Workmentioning

confidence: 99%

State Action Separable Reinforcement Learning

Zhang

Ma²,

Leung

et al. 2020

Preprint

View full text Add to dashboard Cite

Reinforcement Learning (RL) based methods have seen their paramount successes in solving serial decision-making and control problems in recent years. For conventional RL formulations, Markov Decision Process (MDP) and state-action-value function are the basis for the problem modeling and policy evaluation. However, several challenging issues still remain. Among most cited issues, the enormity of state/action space is an important factor that causes inefficiency in accurately approximating the state-action-value function. We observe that although actions directly define the agents' behaviors, for many problems the next state after a state transition matters more than the action taken, in determining the return of such a state transition. In this regard, we propose a new learning paradigm, State Action Separable Reinforcement Learning (sasRL), wherein the action space is decoupled from the value function learning process for higher efficiency. Then, a light-weight transition model is learned to assist the agent to determine the action that triggers the associated state transition. In addition, our convergence analysis reveals that under certain conditions, the convergence time of sasRL is O(T 1/k ), where T is the convergence time for updating the value function in the MDP-based formulation and k is a weighting factor. Experiments on several gaming scenarios show that sasRL outperforms state-of-the-art MDP-based RL algorithms by up to 75%.

show abstract

“…However, the vanilla policy gradient algorithm performs poorly with large discrete action spaces. To address this, we can instead decompose a policy into a component that acts in a latent space of action representations(embeddings) and a component that transforms these representations into actual actions, as shown in [5]. This allows generalization over actions, as similar actions have similar action representations, and improves performance, while speeding up learning.…”

Section: Computing Allocation Strategymentioning

confidence: 99%

Reinforcement Learning for Unified Allocation and Patrolling in Signaling Games with Uncertainty

Venugopal,

Bondi,

Kamarthi

et al. 2020

Preprint

View full text Add to dashboard Cite

Green Security Games (GSGs) have been successfully used in the protection of valuable resources such as fisheries, forests and wildlife. While real-world deployment involves both resource allocation and subsequent coordinated patrolling with communication and realtime, uncertain information, previous game models do not fully address both of these stages simultaneously. Furthermore, adopting existing solution strategies is difficult since they do not scale well for larger, more complex variants of the game models.We therefore first propose a novel GSG model that combines defender allocation, patrolling, real-time drone notification to human patrollers, and drones sending warning signals to attackers. The model further incorporates uncertainty for real-time decisionmaking within a team of drones and human patrollers. Second, we present CombSGPO, a novel and scalable algorithm based on reinforcement learning, to compute a defender strategy for this game model. CombSGPO performs policy search over a multidimensional, discrete action space to compute an allocation strategy that is best suited to a best-response patrolling strategy for the defender, learnt by training a multi-agent Deep Q-Network. We show via experiments that CombSGPO converges to better strategies and is more scalable than comparable approaches. Third, we provide a detailed analysis of the coordination and signaling behavior learnt by CombSGPO, showing group formation between defender resources and patrolling formations based on signaling and notifications between resources. Importantly, we find that strategic signaling emerges in the final learnt strategy. Finally, we perform experiments to evaluate these strategies under different levels of uncertainty.

show abstract

Learning Action Representations for Reinforcement Learning

Cited by 10 publications

References 0 publications

Estimating Q(s,s') with Deep Deterministic Dynamics Gradients

Estimating Q(s,s') with Deep Deterministic Dynamics Gradients

State Action Separable Reinforcement Learning

Reinforcement Learning for Unified Allocation and Patrolling in Signaling Games with Uncertainty

Contact Info

Product

Resources

About