Episodic Memory Deep Q-Networks

Lin, Zichuan; Yang, Guangwen; Zhang, Lintao

doi:10.48550/arxiv.1805.07603

Cited by 11 publications

(21 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Episodic control enables sample-efficiency through explicitly storing the association between returns and state-action pairs in episodic memory [29,4,36]. When combined with Q-learning (habitual control), the episodic memory augments the value function with episodic value estimation, which is shown beneficial to guide the RL agent to latch on good policies during early training [30,46,20].…”

Section: Memory-based Controlsmentioning

confidence: 99%

“…We note that our memory writing allows multiple updates to multiple neighbor slots, which is unlike the single-slot update rule [4,36,30]. Here, the written value is the Monte Carlo return collected from t + 1 to the end of the episode…”

Section: Memory Operatorsmentioning

confidence: 99%

“…Using only one or few instances of past experiences to make decisions, EC agents avoid complicated planning computations, exploiting experiences faster than the other two control methods. In hybrid control systems, EC demonstrates excellent performance and better sample efficiency [36,30].…”

Section: Introductionmentioning

confidence: 99%

“…Other episodic memories store individual state-action pairs, acting as the state-action value table in tabular RL, and can generalize to novel states using nearest neighbor approximations [4,36]. Recent works [35,16,30,46] leverage both episodic and habitual learning by combining state-action episodic memories with Q-learning augmented with parametric value functions like Deep Q-Network (DQN; [34]). The combination of the "fast" non-parametric episodic and "slow" parametric value facilitates Complementary Learning Systems (CLS) -a theory posits that the brain relies on both slow learning of distributed representations (neocortex) and fast learning of pattern-separated representations (hippocampus) [31].…”

Section: Introductionmentioning

confidence: 99%

“…Existing episodic RL methods suffer from 3 issues: (a) near-deterministic assumption [4] which is vulnerable to noisy, stochastic or partially observable environments causing ambiguous observations; (b) sample-inefficiency due to storing state-action-value which demands experiencing all actions to make reliable decisions and inadequate memory writings that prevent fast and accurate value propagation inside the memory [4,36]; and finally, (c) assuming fixed combination between episodic and parametric values [30,16] that makes episodic contribution weight unchanged for different observations and requires manual tuning of the weight. We tackle these open issues by designing a novel model that flexibly integrates habitual, model-based and episodic control into a single architecture for RL.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Model-Based Episodic Memory Induces Dynamic Hybrid Controls

Lê¹,

Karimpanal²,

Abdolshah³

et al. 2021

Preprint

View full text Add to dashboard Cite

Episodic control enables sample efficiency in reinforcement learning by recalling past experiences from an episodic memory. We propose a new model-based episodic memory of trajectories addressing current limitations of episodic control. Our memory estimates trajectory values, guiding the agent towards good policies. Built upon the memory, we construct a complementary learning model via a dynamic hybrid control unifying model-based, episodic and habitual learning into a single architecture. Experiments demonstrate that our model allows significantly faster and better learning than other strong reinforcement learning agents across a variety of environments including stochastic and non-Markovian settings.

show abstract

Section: Memory-based Controlsmentioning

confidence: 99%

Section: Memory Operatorsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Model-Based Episodic Memory Induces Dynamic Hybrid Controls

Lê¹,

Karimpanal²,

Abdolshah³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Model-based Reinforcement Learning: A Survey

Moerland

Broekens

Plaat

et al. 2023

FNT in Machine Learning

223

View full text Add to dashboard Cite

Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is an important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This survey is an integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematically cover approaches to dynamics model learning, including challenges like dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. Second, we present a systematic categorization of planning-learning integration, including aspects like: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. After these two sections, we also discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and we cover the potential benefits of model-based RL. Along the way, the survey also draws connections to several related RL fields, like hierarchical RL and transfer

show abstract