Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Abstract

Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model. To this end, we propose Neural Task Graph (NTG) Networks, which use conjugate task graph as the intermediate representation to modularize both the video demonstration and the derived policy. We empirically show NTG achieves inter-task generalization on two complex tasks: Block Stacking in Bul-letPhysics and Object Collection in AI2-THOR. NTG improves data efficiency with visual input as well as achieve strong generalization without the need for dense hierarchical supervision. We further show that similar performance trends hold when applied to real-world data. We show that NTG can effectively predict task structure on the JIGSAWS surgical dataset and generalize to unseen tasks.

Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning

“…(2) How well does our continuous relaxation of Continuous Planner handle the outputs of our Symbol Grounding Networks? We answer these questions by evaluating our method on two task domains: Block Stacking and Object Sorting in BulletPhysics [4]. We compare the proposed framework with alternative formulations of one-shot imitation learning and evaluate the importance of our continuous relaxation.…”

“…Existing one-shot imitation learning methods [2,4,6] parameterize φ(·) as policy models conditioned on demonstrations. While these methods have been shown to generalize to T meta−test , training such policy networks requires a large amount of data in T meta−train because the policy networks need to simultaneously interpret demonstrations and perform tasks.…”

“…We compare the following baselines for the experiments: Neural Task Graph Networks (NTG) [4]. NTG is the closest to our work among the deep learning-based one-shot imitation learning approaches [2,3,6].…”

“…Experimental Setups. We follow the setup of Block Stacking tasks in previous works [4,6], where the goal is to stack the set of 8 blocks into a target configuration. The blocks are 5 cm cubes.…”

“…The x-axis shows the number of training tasks used in T meta−train and the y-axis shows the success rate of the model on tasks in T meta−test given a single demonstration. All of the planningbased methods outperform the policy networks of NTG [4], even though the NTG policy is already parameterized by the task graph to improve the data efficiency. The results show the importance of our formulation of one-shot imitation learning as a planning problem to disentangle the intertask generalization from the task execution.…”