A Unifying Framework for Reinforcement Learning and Planning

Moerland, Thomas M.; Broekens, Joost; Plaat, Aske; Jonker, Catholijn M.

doi:10.48550/arxiv.2006.15009

Cited by 6 publications

(14 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The third crucial consideration is: how to actually plan? Of course, we do not aim to provide a full survey of planning methods here, but refer the reader to Moerland et al (2020a) for a recent framework to categorize planning and RL methods. Instead, we focus on some crucial decisions we have to make for the integration, on a) the use of potential differentiability of the model, b) the direction of planning, c) the breadth and depth of the plan, and d) the way of dealing with uncertainty.…”

Section: How To Plan?mentioning

confidence: 99%

“…Hester and Stone (2012b) gives a book-chapter presentation of model-based RL methods, but their work does not provide a full overview, nor does it incorporate the vast recent literature on neural network approximation in model-based reinforcement learning. Moerland et al (2020a) presents a framework for reinforcement learning and planning that disentangles their common underlying dimensions, but does not focus on their integration. In some sense, Moerland et al (2020a) looks 'inside' each planning or reinforcement learning cycle, strapping their shared algorithmic space down into its underlying dimensions.…”

Section: Related Workmentioning

confidence: 99%

“…Moerland et al (2020a) presents a framework for reinforcement learning and planning that disentangles their common underlying dimensions, but does not focus on their integration. In some sense, Moerland et al (2020a) looks 'inside' each planning or reinforcement learning cycle, strapping their shared algorithmic space down into its underlying dimensions. Instead, our work looks 'over' the planning cycle, focusing on how we may integrate planning, learning and acting to provide mutual benefit.…”

Section: Related Workmentioning

confidence: 99%

“…The key difference between planning and learning is whether a model of the environment dynamics is known (planning) or unknown (reinforcement learning). A framework to describe the commonalities between planning and reinforcement learning is provided by Moerland et al (2020a).…”

Section: Introductionmentioning

confidence: 99%

“…Even model-based RL methods with a known model, like AlphaGo Zero(Silver et al, 2017a), sometimes voluntarily introduce the notion of a real environment and current state. The real environment step introduces a form of pruning, as it ensures that we move forward at some point, obtaining information about deeper nodes (seeMoerland et al (2020a) as well).…”

mentioning

confidence: 99%

See 4 more Smart Citations

Model-based Reinforcement Learning: A Survey

Moerland¹,

Broekens²,

Plaat³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is a key challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This paper presents a survey of the integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematically cover approaches to dynamics model learning, including challenges like dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. Second, we present a systematic categorization of planning-learning integration, including aspects like: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. After these two key sections, we also discuss the potential benefits of model-based RL, like enhanced data efficiency, targeted exploration, and improved stability. Along the survey, we also draw connections to several related RL fields, like hierarchical RL and transfer, and other research disciplines, like behavioural psychology. Altogether, the survey presents a broad conceptual overview of planning-learning combinations for MDP optimization.

show abstract

Section: How To Plan?mentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Model-based Reinforcement Learning: A Survey

Moerland¹,

Broekens²,

Plaat³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Model-based Reinforcement Learning: A Survey

Moerland

Broekens

Plaat

et al. 2023

FNT in Machine Learning

223

View full text Add to dashboard Cite

Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is an important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This survey is an integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematically cover approaches to dynamics model learning, including challenges like dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. Second, we present a systematic categorization of planning-learning integration, including aspects like: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. After these two sections, we also discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and we cover the potential benefits of model-based RL. Along the way, the survey also draws connections to several related RL fields, like hierarchical RL and transfer

show abstract