Hierarchical Few-Shot Imitation with Skill Transition Models

Hakhamaneshi, Kourosh; Zhao, Ruihan; Zhan, Albert; Abbeel, Pieter; Laskin, Michael

doi:10.48550/arxiv.2107.08981

Cited by 4 publications

(7 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, [17] shows the capability to extract the skills abstraction from demonstration dataset. Skills priors based methods [25][26] [27], similar to options discovering, extract the skill representation from demonstrations dataset Fig. 1: Fast Imitation and Policy Reuse Learning.…”

Section: Related Work a Hierarchical Reinforcement Learning Temporal ...mentioning

confidence: 99%

“…[10] discusses general policy improvement and general policy evaluation using successor features [11] [12]. One-shot and Few-shot learning are similar topics related to fast learning that need some examples [28] [27]. [28] gives convincing results upon one-shot imitation learning using context and attention extraction.…”

Section: B One-shot Learning Fast Learning Imitation Learningmentioning

confidence: 99%

“…Different from our work, it requires a large number of demonstration datasets for training. Similar to us, FIST [27] proposes few-shot imitation learning, but our work does not involve the complex networks setup and skills embedding learning in FIST. In multi-stage, long-horizon and reward-sparse tasks, imitation learning methods [22][29] play an integral role in increasing learning efficiency.…”

Section: B One-shot Learning Fast Learning Imitation Learningmentioning

confidence: 99%

See 2 more Smart Citations

FIRL: Fast Imitation and Policy Reuse Learning

Chen¹,

Zhang²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Intelligent robotics policies have been widely researched for challenging applications such as opening doors, washing dishes, and table organization. We refer to a "Policy Pool", containing skills that be easily accessed and reused. There are researches to leverage the pool, such as policy reuse, modular learning, assembly learning, transfer learning, hierarchical reinforcement learning (HRL), etc. However, most methods generally do not perform well in learning efficiency and require large datasets for training. This work focuses on enabling fast learning based on the policy pool. It should learn fast enough in one-shot or few-shot by avoiding learning from scratch. We also allow it to interact and learn from humans, but the training period should be within minutes. We propose FIRL, Fast (one-shot) Imitation, and Policy Reuse Learning. Instead of learning a new skill from scratch, it performs the one-shot imitation learning on the higher layer under a 2-layer hierarchical mechanism. Our method reduces a complex task learning to a simple regression problem that it could solve in a few offline iterations. The agent could have a good command of a new task given a one-shot demonstration. We demonstrate this method on the OpenDoors mini-grid environment, and the code is available on http://www.github.com/yiwc/firl.

show abstract

Section: Related Work a Hierarchical Reinforcement Learning Temporal ...mentioning

confidence: 99%

Section: B One-shot Learning Fast Learning Imitation Learningmentioning

confidence: 99%

Section: B One-shot Learning Fast Learning Imitation Learningmentioning

confidence: 99%

See 1 more Smart Citation

FIRL: Fast Imitation and Policy Reuse Learning

Chen¹,

Zhang²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While our procedure observations can appear to be high-level labels or sub-goals similar to [37,38], we do not make assumptions about the structure of procedure observations, which can simply be scalar variable values computed during program execution. There is also a large body of literature that assumes access to a suboptimal offline dataset in addition to the expert demonstration data collected from the same environment, and conduct representation learning [39,40,41,42,43], hierarchical skill extraction [44,45,46], or dynamics model learning [47,48,49] on the suboptimal offline data followed by imitation learning from an expert. These works commonly assume good coverage in the offline data, which requires large amounts of state-action pairs being collected from running additional policies in the environment.…”

Section: Related Workmentioning

confidence: 99%

Chain of Thought Imitation with Procedure Cloning

Yang¹,

Schuurmans²,

Abbeel³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Imitation learning aims to extract high-performance policies from logged demonstrations of expert behavior. It is common to frame imitation learning as a supervised learning problem in which one fits a function approximator to the input-output mapping exhibited by the logged demonstrations (input observations to output actions). While the framing of imitation learning as a supervised input-output learning problem allows for applicability in a wide variety of settings, it is also an overly simplistic view of the problem in situations where the expert demonstrations provide much richer insight into expert behavior. For example, applications such as path navigation, robot manipulation, and strategy games acquire expert demonstrations via planning, search, or some other multi-step algorithm, revealing not just the output action to be imitated but also the procedure for how to determine this action. While these intermediate computations may use tools not available to the agent during inference (e.g., environment simulators), they are nevertheless informative as a way to explain an expert's mapping of state to actions. To properly leverage expert procedure information without relying on the privileged tools the expert may have used to perform the procedure, we propose procedure cloning, which applies supervised sequence prediction to imitate the series of expert computations. This way, procedure cloning learns not only what to do (i.e., the output action), but how and why to do it (i.e., the procedure). Through empirical analysis on navigation, simulated robotic manipulation, and game-playing environments, we show that imitating the intermediate computations of an expert's behavior enables procedure cloning to learn policies exhibiting significant generalization to unseen environment configurations, including those configurations for which running the expert's procedure directly is infeasible. 1

show abstract

“…Few approaches for robotic manipulation have a dedi-cated planning component trainable separately from a controller. One exception are approaches that plan in a skill space [21,35,38,45,50], which PLEX can be modified to do as well. Conceptually, PLEX falls under the paradigm of learning from observations (LfO), but existing LfO approaches don't have multitask zero-shot planning capability [3,36,41,42] or demostrate it only in lowdimensional environments across similar tasks [51].…”

Section: Introductionmentioning

confidence: 99%

PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining

Thomas¹,

Cheng²,

Loynd³

et al. 2023

Preprint

View full text Add to dashboard Cite

A rich representation is key to general robotic manipulation, but existing model architectures require a lot of data to learn it. Unfortunately, ideal robotic manipulation training data, which comes in the form of expert visuomotor demonstrations for a variety of annotated tasks, is scarce. In this work we propose PLEX, a transformer-based architecture that learns from task-agnostic visuomotor trajectories accompanied by a much larger amount of taskconditioned object manipulation videos -a type of robotics-relevant data available in quantity.The key insight behind PLEX is that the trajectories with observations and actions help induce a latent feature space and train a robot to execute task-agnostic manipulation routines, while a diverse set of video-only demonstrations can efficiently teach the robot how to plan in this feature space for a wide variety of tasks. In contrast to most works on robotic manipulation pretraining, PLEX learns a generalizable sensorimotor multi-task policy, not just an observational representation. We also show that using relative positional encoding in PLEX's transformers further increases its data efficiency when learning from human-collected demonstrations. Experiments showcase PLEX's generalization on Meta-World benchmark and establish state-of-the-art performance in challenging Robosuite environments.

show abstract

Hierarchical Few-Shot Imitation with Skill Transition Models

Cited by 4 publications

References 4 publications

FIRL: Fast Imitation and Policy Reuse Learning

FIRL: Fast Imitation and Policy Reuse Learning

Chain of Thought Imitation with Procedure Cloning

PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining

Contact Info

Product

Resources

About