Integrating Sample-Based Planning and Model-Based Reinforcement Learning

Abstract

Recent advancements in model-based reinforcement learning have shown that the dynamics of many structured domains (e.g. DBNs) can be learned with tractable sample complexity, despite their exponentially large state spaces. Unfortunately, these algorithms all require access to a planner that computes a near optimal policy, and while many traditional MDP algorithms make this guarantee, their computation time grows with the number of states. We show how to replace these over-matched planners with a class of sample-based planners — whose computation time is independent of the number of states — without sacrificing the sample-efficiency guarantees of the overall learning algorithms. To do so, we define sufficient criteria for a sample-based planner to be used in such a learning system and analyze two popular sample-based approaches from the literature. We also introduce our own sample-based planner, which combines the strategies from these algorithms and still meets the criteria for integration into our learning system. In doing so, we define the first complete RL solution for compactly represented (exponentially sized) state spaces with efficiently learnable dynamics that is both sample efficient and whose computation time does not grow rapidly with the number of states.

Bootstrapping Simulation-Based Algorithms with a Suboptimal Policy

“…Proof sketch. The proof is identical to that of Proposition 3 by Walsh et al (Walsh, Goschin, and Littman 2010), by noting that every trial of FSSS-Aux has to end at a node with zero bound gap. If such a node is not a leaf, it would not be selected in the first place, because it cannot be the node with widest bound gap.…”

“…FSSS is the latest successor of SS that is able to combine the performance guarantee of SS with selective sampling (Walsh, Goschin, and Littman 2010). Instead of directly forming point estimates of the Q-and V-values in a look-ahead tree, it maintains interval estimates that allow it then to guide the sampling to the promising and/or unexplored branches of the look-ahead tree based on the widths and upper bounds of the intervals.…”

“…In this paper, we investigate the auxiliary estimation approach in other more conservative simulation-based algorithms for large MDPs. In particular, we examine the Aux-method coupled with Sparse Sampling (SS) (Kearns, Mansour, and Ng 2002), the backbone of many modern simulation-based algorithms, and Forward Search Sparse Sampling (FSSS) (Walsh, Goschin, and Littman 2010), one of the state-of-the-art sparse sampling algorithms. While there have been various efforts in enhancing the performance of UCT (Gelly and Silver 2007;Finnsson and Björnsson 2008;Chaslot et al 2010;Nguyen, Lee, and Leong 2012;Keller and Helmert 2013), to the best of our knowledge, our work is among the first to investigate possible enhancement for the algorithms addressed.…”

What Can I Not Do? Towards an Architecture for Reasoning about and Learning Affordances

“…Interactive learning has also been posed as an RL problem with an underlying MDP (Sutton and Barto 1998). Approaches for efficient RL in dynamic domains include sample-based planning algorithms (Walsh, Goschin, and Littman 2010), and Relational RL, which uses relational representations and regression for Q-function generalization (Dzeroski, Raedt, and Driessens 2001;Tadepalli, Givan, and Driessens 2004). However, most RRL algorithms focus on planning, limit generalization to a single planning task, or do not support the desired commonsense reasoning capabilities.…”

Heuristic Q-learning based on experience replay for three-dimensional path planning of the unmanned aerial vehicle

“…Andrew et al 7 introduced shaping function into reinforcement learning and added heuristic value to the returns of agents, which effectively improved the convergence speed. Asmuth et al 8 used potential field function as a priori knowledge to enlighten the reinforcement learning process and proved the effectiveness of the algorithm.…”