Coarse-to-fine Q-attention with Tree Expansion

James, S. Jill; Abbeel, Pieter

doi:10.48550/arxiv.2204.12471

Cited by 4 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following James and Davison [39], we fill a replay buffer with 50 or 100 expert demonstrations. Unlike prior approaches that utilize path planner with the policy to output next best gripper pose [39,3,21,40,41], our RL agent outputs relative change in gripper position. We provide further details in Appendix A. : Aggregate success rate on five multi-view and single-view control tasks and two viewpointrobust control tasks.…”

Section: Methodsmentioning

confidence: 99%

Multi-View Masked World Models for Visual Robotic Manipulation

Seo¹,

Kim²,

James³

et al. 2023

Preprint

View full text Add to dashboard Cite

Visual robotic manipulation research and applications often use multiple cameras, or views, to better perceive the world. How else can we utilize the richness of multi-view data? In this paper, we investigate how to learn good representations with multi-view data and utilize them for visual robotic manipulation. Specifically, we train a multi-view masked autoencoder which reconstructs pixels of randomly masked viewpoints and then learn a world model operating on the representations from the autoencoder. We demonstrate the effectiveness of our method in a range of scenarios, including multi-view control and single-view control with auxiliary cameras for representation learning. We also show that the multi-view masked autoencoder trained with multiple randomized viewpoints enables training a policy with strong viewpoint randomization and transferring the policy to solve real-robot tasks without camera calibration and an adaptation procedure. Video demonstrations in real-world experiments and source code are available at the project website: https://sites.google.com/view/mv-mwm.

show abstract

Section: Methodsmentioning

confidence: 99%

Multi-View Masked World Models for Visual Robotic Manipulation

Seo¹,

Kim²,

James³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Akbulut et al [2] introduce a new framework called Adaptive Conditional Neural Movement Primitives, combining supervised learning and RL to conserve old skills learned from robot demonstrations while being adaptive to new environments. James and Davison [51] present a coarse-to-fine discrete RL algorithm to solve sparse reward manipulation tasks by using only a small amount of demonstration and exploration data (work extended by [49] and [50]). Celemin et al [19] include human corrective advice in the action domain through a learning-from-demonstration approach, while an RL algorithm guides the learning process by filtering out human feedback that does not maximize the reward.…”

Section: Learning From Demonstrationmentioning

confidence: 99%

Guided Reinforcement Learning: A Review and Evaluation for Efficient and Effective Real-World Robotics [Survey]

Eßer

Bach

Jestel

et al. 2023

IEEE Robot. Automat. Mag.

View full text Add to dashboard Cite

ecent successes aside, reinforcement learning (RL) still faces significant challenges in its application to the real-world robotics domain. Guiding the learning process with additional knowledge offers a potential solution, thus leveraging the strengths of data-and knowledge-driven approaches. However, this field of research encompasses several disciplines and hence would benefit from a structured overview.In this article, we propose a concept of guided RL that provides a systematic approach toward accelerating the training process and improving performance for real-world robotics settings. We introduce a taxonomy that structures guided RL approaches and shows how different sources of knowledge can be integrated into the learning pipeline in a practical way. Based on this, we describe available approaches in this field and quantitatively evaluate their specific impact in terms of efficiency, effectiveness, and sim-to-real transfer within the robotics domain. However, learning control policies in such a way naturally requires many interactions with the environment. This emphasizes the importance of both collecting highquality samples and exploring the search space in a sample-efficient manner. While directly learning on real robots is appealing, it comes along with substantial challenges, such as high sample cost, partial observability, and safety constraints [28]. Hence, simulators are often

show abstract

“…C2F-ARM takes this next-best pose and uses a motion planner to take the robot to the goal pose. In this work, we use the original C2F-ARM algorithm, and do not include any of its subsequent extensions, e.g., learned path ranking [66] and tree expansion [67].…”

Section: Additional Rlbench Remarksmentioning

confidence: 99%

On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning

Zhao¹,

Abbeel²,

James³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Intelligent agents should have the ability to leverage knowledge from previously learned tasks in order to learn new ones quickly and efficiently. Meta-learning approaches have emerged as a popular solution to achieve this. However, metareinforcement learning (meta-RL) algorithms have thus far been restricted to simple environments with narrow task distributions. Moreover, the paradigm of pretraining followed by fine-tuning to adapt to new tasks has emerged as a simple yet effective solution in supervised and self-supervised learning. This calls into question the benefits of meta-learning approaches in reinforcement learning, which typically come at the cost of high complexity. We therefore investigate meta-RL approaches in a variety of vision-based benchmarks, including Procgen, RLBench, and Atari, where evaluations are made on completely novel tasks. Our findings show that when meta-learning approaches are evaluated on different tasks (rather than different variations of the same task), multi-task pretraining with fine-tuning on new tasks performs equally as well, or better, than meta-pretraining with meta test-time adaptation. This is encouraging for future research, as multi-task pretraining tends to be simpler and computationally cheaper than meta-RL. From these findings, we advocate for evaluating future meta-RL methods on more challenging tasks and including multi-task pretraining with fine-tuning as a simple, yet strong baseline. 1

show abstract

Coarse-to-fine Q-attention with Tree Expansion

Cited by 4 publications

References 20 publications

Multi-View Masked World Models for Visual Robotic Manipulation

Multi-View Masked World Models for Visual Robotic Manipulation

Guided Reinforcement Learning: A Review and Evaluation for Efficient and Effective Real-World Robotics [Survey]

On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning

Contact Info

Product

Resources

About