Imagination-Augmented Agents for Deep Reinforcement Learning

Weber, Théophane; Racanière, Sébastien; Reichert, David P.; Buesing, Lars; Guez, Arthur; Rezende, Danilo Jimenez; Badia, Adrià Puigdomènech; Vinyals, Oriol; Heess, Nicolas; Li, Yujia; Pascanu, Razvan; Battaglia, Peter W.; Hassabis, Demis; Silver, David; Wierstra, Daan

doi:10.48550/arxiv.1707.06203

Cited by 43 publications

(51 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results provide a theoretical foundation for the optimality of deep imagination in model-based planning by showing that it becomes the dominant strategy in one-shot allocations of resources over a broad range of capacity and environmental parameters. Recent deep-learning work has studied through numerical simulations how agents can benefit from imagining future steps by using models of the environment [46,47,48,49], and thus our results might help to clarify and stress the importance of deep tree sampling through mental simulations of state transitions.…”

Section: Discussionmentioning

confidence: 83%

Deep imagination is a close to optimal policy for planning in large decision trees under limited resources

Moreno‐Bote¹,

Chiara²

2021

Preprint

View full text Add to dashboard Cite

Many decisions involve choosing an uncertain course of actions in deep and wide decision trees, as when we plan to visit an exotic country for vacation. In these cases, exhaustive search for the best sequence of actions is not tractable due to the large number of possibilities and limited time or computational resources available to make the decision. Therefore, planning agents need to balance breadth (exploring many actions at each level of the tree) and depth (exploring many levels in the tree) to allocate optimally their finite search capacity. We provide efficient analytical solutions and numerical analysis to the problem of allocating finite sampling capacity in one shot to large decision trees. We find that in general the optimal policy is to allocate few samples per level so that deep levels can be reached, thus favoring depth over breadth search. In contrast, in poor environments and at low capacity, it is best to broadly sample branches at the cost of not sampling deeply, although this policy is marginally better than deep allocations. Our results provide a theoretical foundation for the optimality of deep imagination for planning and show that it is a generally valid heuristic that could have evolved from the finite constraints of cognitive systems.

show abstract

Section: Discussionmentioning

confidence: 83%

Deep imagination is a close to optimal policy for planning in large decision trees under limited resources

Moreno‐Bote¹,

Chiara²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Many models are proposed to solve Sokoban. Both modelbased methods [9,10,21], as well as model-free methods can reach competitive performance [8]. Curriculum learning has been used to solve a difficult Sokoban instance [6].…”

Section: Sokobanmentioning

confidence: 99%

Potential-based Reward Shaping in Sokoban

Yang,

Preuss,

Plaat

2021

Preprint

View full text Add to dashboard Cite

Learning to solve sparse-reward reinforcement learning problems is difficult, due to the lack of guidance towards the goal. But in some problems, prior knowledge can be used to augment the learning process. Reward shaping is a way to incorporate prior knowledge into the original reward function in order to speed up the learning. While previous work has investigated the use of expert knowledge to generate potential functions, in this work, we study whether we can use a search algorithm(A*) to automatically generate a potential function for reward shaping in Sokoban, a well-known planning task. The results showed that learning with shaped reward function is faster than learning from scratch. Our results indicate that distance functions could be a suitable function for Sokoban. This work demonstrates the possibility of solving multiple instances with the help of reward shaping. The result can be compressed into a single policy, which can be seen as the first phrase towards training a general policy that is able to solve unseen instances. 4

show abstract

“…Trajectories from a learned model are also used as extra inputs for a value function (Weber et al, 2017), which reduces the negative influence of the model prediction error. In this paper, we focus on the simplest Dyna-style planning and leave the combination of RA and more advanced planning techniques for future work.…”

Section: Related Workmentioning

confidence: 99%

Deep Residual Reinforcement Learning

Zhang,

Boehmer,

Whiteson

2019

Preprint

View full text Add to dashboard Cite

We revisit residual algorithms in both model-free and model-based reinforcement learning settings. We propose the bidirectional target network technique to stabilize residual algorithms, yielding a residual version of DDPG that significantly outperforms vanilla DDPG in the DeepMind Control Suite benchmark. Moreover, we find the residual algorithm an effective approach to the distribution mismatch problem in model-based planning. Compared with the existing TD(k) method, our residual-based method makes weaker assumptions about the model and yields a greater performance boost.Preprint. Under review.

show abstract

Imagination-Augmented Agents for Deep Reinforcement Learning

Cited by 43 publications

References 28 publications

Deep imagination is a close to optimal policy for planning in large decision trees under limited resources

Deep imagination is a close to optimal policy for planning in large decision trees under limited resources

Potential-based Reward Shaping in Sokoban

Deep Residual Reinforcement Learning

Contact Info

Product

Resources

About