Hierarchical Deep Q-Network from Imperfect Demonstrations in Minecraft

Skrynnik, Alexey; Staroverov, Aleksey; Aitygulov, Ermek; Aksenov, Kirill; Davydov, Vasilii; Panov, Aleksandr I.

doi:10.48550/arxiv.1912.08664

Cited by 2 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the task of obtaining diamond in MineRL (hierarhical set), we demonstrate the ForgER ability to use extracted subtask graph G and use it for goal oriented ogranization of experience replay and data augmentation. In a crucial experiment, we compare our ForgER approach, ForgER++ modication (a heuristically modified hierarchy of subtasks G) with the best MineRL competition solution [26].…”

Section: Hierarchical Settingmentioning

confidence: 99%

See 1 more Smart Citation

Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations

Skrynnik¹,

Staroverov²,

Aitygulov³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Currently, deep reinforcement learning (RL) shows impressive results in complex gaming and robotic environments. Often these results are achieved at the expense of huge computational costs and require an incredible number of episodes of interaction between the agent and the environment. There are two main approaches to improving the sample efficiency of reinforcement learning methods -using hierarchical methods and expert demonstrations. In this paper, we propose a combination of these approaches that allow the agent to use low-quality demonstrations in complex vision-based environments with multiple related goals. Our forgetful experience replay (ForgER) algorithm effectively handles errors in expert data and reduces quality losses when adapting the action space and states representation to the agent's capabilities. Our proposed goal-oriented structuring of replay buffer allows the agent to automatically highlight sub-goals for solving complex hierarchical tasks in demonstrations. Our method is universal and can be integrated into various off-policy methods. It surpasses all known existing state-of-the-art RL methods using expert demonstrations on various model environments. The solution based on our algorithm beats all the solutions for the famous MineRL competition and allows the agent to mine a diamond in the Minecraft environment.Preprint. Under review.

show abstract

Section: Hierarchical Settingmentioning

confidence: 99%

“…In this regard, we have applied our method in the well-known complex Minecraft environment, which has recently served as a good benchmark for testing RL algorithms in rich hierarchical settings [7,25]. This allowed us to demonstrate the performance of ForgER and surpass the results of recent SOTA methods in MineRL competition [26].…”

Section: Introductionmentioning

confidence: 99%

Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations

Skrynnik¹,

Staroverov²,

Aitygulov³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The top team, CDS, used a hierarchical deep Q-network with forgetting that uses an adaptive ratio for sampling expert demonstrations from a separate demonstration replay buffer 13 (Skrynnik et al, 2019). The second-place team, mc rl, trained their hierarchical policies entirely from human demonstrations with no environment interactions 14 .…”

Section: Summary Of Top Nine Solutionsmentioning

confidence: 99%

Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning

Milani,

Topin,

Houghton

et al. 2020

Preprint

View full text Add to dashboard Cite

To facilitate research in the direction of sample-efficient reinforcement learning, we held the MineRL Competition on Sample-Efficient Reinforcement Learning Using Human Priors at the Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2019). The primary goal of this competition was to promote the development of algorithms that use human demonstrations alongside reinforcement learning to reduce the number of samples needed to solve complex, hierarchical, and sparse environments. We describe the competition and provide an overview of the top solutions, each of which use deep reinforcement learning and/or imitation learning. We also discuss the impact of our organizational decisions on the competition as well as future directions for improvement.

show abstract

Hierarchical Deep Q-Network from Imperfect Demonstrations in Minecraft

Cited by 2 publications

References 0 publications

Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations

Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations

Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning

Contact Info

Product

Resources

About