One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Video

Goo, Wonjoon; Niekum, Scott

doi:10.48550/arxiv.1806.11244

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While, third-person imitation learning uses date from other agents or viewpoints [27,35]. Recent methods for one-shot imitation learning [8,11,13,40,41,42] can translate a single demonstration to an executable pol- icy. The most similar to ours is NTP [41] that also learns long-horizon tasks.…”

Section: Related Workmentioning

confidence: 99%

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Huang

Nair

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

104

100

View full text Add to dashboard Cite

Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model. To this end, we propose Neural Task Graph (NTG) Networks, which use conjugate task graph as the intermediate representation to modularize both the video demonstration and the derived policy. We empirically show NTG achieves inter-task generalization on two complex tasks: Block Stacking in Bul-letPhysics and Object Collection in AI2-THOR. NTG improves data efficiency with visual input as well as achieve strong generalization without the need for dense hierarchical supervision. We further show that similar performance trends hold when applied to real-world data. We show that NTG can effectively predict task structure on the JIGSAWS surgical dataset and generalize to unseen tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Huang

Nair

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

104

100

View full text Add to dashboard Cite

show abstract

“…Realworld tasks are often long-horizon and multi-step, posing severe challenges for simple techniques such as behavior cloning-based methods [2]. Recent works in one-shot imitation learning aim to mitigate this challenge by imposing modular or hierarchical task structures in order to learn reusable subtask policies [4,6,[11][12][13]. NTP [6] decomposes demonstration with hierarchical programs.…”

Section: Related Workmentioning

confidence: 99%

Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning

Huang

Zhu

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

We address one-shot imitation learning, where the goal is to execute a previously unseen task based on a single demonstration. While there has been exciting progress in this direction, most of the approaches still require a few hundred tasks for meta-training, which limits the scalability of the approaches. Our main contribution is to formulate one-shot imitation learning as a symbolic planning problem along with the symbol grounding problem. This formulation disentangles the policy execution from the inter-task generalization and leads to better data efficiency. The key technical challenge is that the symbol grounding is prone to error with limited training data and leads to subsequent symbolic planning failures. We address this challenge by proposing a continuous relaxation of the discrete symbolic planner that directly plans on the probabilistic outputs of the symbol grounding model. Our continuous relaxation of the planner can still leverage the information contained in the probabilistic symbol grounding and significantly improve over the baseline planner for the oneshot imitation learning tasks without using large training data.

show abstract

One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Video

Cited by 2 publications

References 0 publications

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning

Contact Info

Product

Resources

About