Visual Robot Task Planning

Paxton, Chris; Barnoy, Yotam; Katyal, Kapil D.; Arora, Raman; Hager, Gregory D.

doi:10.1109/icra.2019.8793736

Cited by 50 publications

(44 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our aim is to extend these methods to enable improvisational tool use. Visual MPC, as well as other video prediction-based planners [9,16,36,46], generally do not succeed at such temporally extended tasks. To this end, we propose to incorporate demonstrations into the algorithm to enable multi-stage tooluse capabilities, while still retaining the flexibility of goaldirected planning to accomplish varied user-specified goals.…”

Section: Related Workmentioning

confidence: 99%

Improvisation through Physical Understanding: Using Novel Objects As Tools with Visual Foresight

Xie¹,

Ebert²,

Levine³

et al. 2019

Robotics: Science and Systems XV

View full text Add to dashboard Cite

Machine learning techniques have enabled robots to learn narrow, yet complex tasks and also perform broad, yet simple skills with a wide variety of objects. However, learning a model that can both perform complex tasks and generalize to previously unseen objects and goals remains a significant challenge. We study this challenge in the context of "improvisational" tool use: a robot is presented with novel objects and a user-specified goal (e.g., sweep some clutter into the dustpan), and must figure out, using only raw image observations, how to accomplish the goal using the available objects as tools. We approach this problem by training a model with both a visual and physical understanding of multi-object interactions, and develop a sampling-based optimizer that can leverage these interactions to accomplish tasks. We do so by combining diverse demonstration data with self-supervised interaction data, aiming to leverage the interaction data to build generalizable models and the demonstration data to guide the model-based RL planner to solve complex tasks. Our experiments show that our approach can solve a variety of complex tool use tasks from raw pixel inputs, outperforming both imitation learning and self-supervised learning individually. Furthermore, we show that the robot can perceive and use novel objects as tools, including objects that are not conventional tools, while also choosing dynamically to use or not use tools depending on whether or not they are required. Videos of the results are available online 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

Improvisation through Physical Understanding: Using Novel Objects As Tools with Visual Foresight

Xie¹,

Ebert²,

Levine³

et al. 2019

Robotics: Science and Systems XV

View full text Add to dashboard Cite

show abstract

“…Simultaneously, prospection has been used before in RL, often as a means of model-based control [19], [17], [20], and for fine control tasks like cutting [19]. In addition, our approach is compatible with work in Visual Robot Task Planning [4], which shows that prospective subgoal predictions can be used to generate task plans.…”

Section: Related Workmentioning

confidence: 92%

“…Our work draws inspiration from recent efforts on learning abstract representations for planning [4]. We build primarily on work in planning and natural language processing, with important future work in manipulation.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Prospection: Interpretable plans from language by predicting the future

Paxton

Bisk

Thomason

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

High-level human instructions often correspond to behaviors with multiple implicit steps. In order for robots to be useful in the real world, they must be able to to reason over both motions and intermediate goals implied by human instructions. In this work, we propose a framework for learning representations that convert from a natural-language command to a sequence of intermediate goals for execution on a robot. A key feature of this framework is prospection, training an agent not just to correctly execute the prescribed command, but to predict a horizon of consequences of an action before taking it. We demonstrate the fidelity of plans generated by our framework when interpreting real, crowd-sourced natural language commands for a robot in simulated scenes.

show abstract

“…al. [37] learn transitions and an action value function in the latent space, and use that to produce visual plans on simulated domains. To our knowledge, we present the first application of plannable features for real robot experiments.…”

Section: Related Workmentioning

confidence: 99%

Learning Robotic Manipulation through Visual Planning and Acting

Wang¹,

Kurutach²,

Liu³

et al. 2019

Robotics: Science and Systems XV

View full text Add to dashboard Cite

Planning for robotic manipulation requires reasoning about the changes a robot can affect on objects. When such interactions can be modelled analytically, as in domains with rigid objects, efficient planning algorithms exist. However, in both domestic and industrial domains, the objects of interest can be soft, or deformable, and hard to model analytically. For such cases, we posit that a data-driven modelling approach is more suitable. In recent years, progress in deep generative models has produced methods that learn to 'imagine' plausible images from data. Building on the recent Causal InfoGAN generative model, in this work we learn to imagine goal-directed object manipulation directly from raw image data of self-supervised interaction of the robot with the object. After learning, given a goal observation of the system, our model can generate an imagined plan -a sequence of images that transition the object into the desired goal. To execute the plan, we use it as a reference trajectory to track with a visual servoing controller, which we also learn from the data as an inverse dynamics model. In a simulated manipulation task, we show that separating the problem into visual planning and visual tracking control is more sample efficient and more interpretable than alternative datadriven approaches. We further demonstrate our approach on learning to imagine and execute in 3 environments, the final of which is deformable rope manipulation on a PR2 robot.

show abstract

Visual Robot Task Planning

Cited by 50 publications

References 24 publications

Improvisation through Physical Understanding: Using Novel Objects As Tools with Visual Foresight

Improvisation through Physical Understanding: Using Novel Objects As Tools with Visual Foresight

Prospection: Interpretable plans from language by predicting the future

Learning Robotic Manipulation through Visual Planning and Acting

Contact Info

Product

Resources

About