Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos

Chen, Annie S.; Nair, Satish; Finn, Chelsea

doi:10.48550/arxiv.2103.16817

Cited by 7 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Related to challenges of generalisation surrounding reward functions, for many real-world problems designing good reward functions is very difficult. A promising approach is using inverse RL [168,169,IRL] to learn a reward function from human demonstrations [170,171,172,173], rather than hand-crafting reward functions for each task. This is often more time-efficient, as demonstrating a task is easier than specifying a reward function for it.…”

Section: Real World Reinforcement Learning Generalisationmentioning

confidence: 99%

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Kirk¹,

Zhang²,

Grefenstette³

et al. 2021

Preprint

View full text Add to dashboard Cite

The study of generalisation in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We provide a unifying formalism and terminology for discussing different generalisation problems, building upon previous works. We go on to categorise existing benchmarks for generalisation, as well as current methods for tackling the generalisation problem. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in generalisation, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for generalisation, and we recommend building benchmarks in underexplored problem settings such as offline RL generalisation and reward-function variation.

show abstract

Section: Real World Reinforcement Learning Generalisationmentioning

confidence: 99%

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Kirk¹,

Zhang²,

Grefenstette³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, these require a large amount of training data. Recent works [5], [6] leverage action recognition models, such as the action classifiers trained on the 20BN Something-Something dataset [29], to identify whether the robot is performing the desired task. However, while these classifiers are useful for identifying the class of motions for short interactions, we show that they do not retain enough information to analyze tasks with longer duration and multiple repetitions.…”

Section: Learning From Human Demonstrationsmentioning

confidence: 99%

“…In this work, we address how robots can appropriately represent and learn periodic policies by watching humans. While prior works considered learning manipulation skills from human demonstrations [1]- [6], less attention has been given to periodic tasks. These tasks repeat similar motion with only small differences between repetitions.…”

Section: Introductionmentioning

confidence: 99%

Learning Periodic Tasks from Human Demonstrations

Yang¹,

Zhang²,

Settle³

et al. 2021

Preprint

View full text Add to dashboard Cite

We develop a method for learning periodic tasks from visual demonstrations. The core idea is to leverage periodicity in the policy structure to model periodic aspects of the tasks. We use active learning to optimize parameters of rhythmic dynamic movement primitives (rDMPs) and propose an objective to maximize the similarity between the motion of objects manipulated by the robot and the desired motion in human video demonstrations. We consider tasks with deformable objects and granular matter whose state is challenging to represent and track: wiping surfaces with a cloth, winding cables/wires, stirring granular matter with a spoon. Our method does not require tracking markers or manual annotations. The initial training data consists of 10-minute videos of random unpaired interactions with objects by the robot and human. We use these for unsupervised learning of a keypoint model to get task-agnostic visual correspondences. Then, we use Bayesian optimization to optimize rDMPs from a single human video demonstration within few robot trials. We present simulation and hardware experiments to validate our approach.

show abstract

“…and task learning that can generalize across multiple environments (How should I do it?). Recent works have just started to explored the use of pre-existing foundational models from language and vision towards robotics [9]- [13], and also the development of robotics-specific foundational models [8,14].…”

Section: Introductionmentioning

confidence: 99%

Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers

Bucker¹,

Figueredo²,

Haddadin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible languagebased interface for human-robot collaboration, which allows a user to reshape existing trajectories for an autonomous agent. We take advantage of recent advancements in the field of large language models (BERT and CLIP) to encode the user command, and then combine these features with trajectory information using multi-modal attention transformers. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands, and treat the trajectory generation process as a sequence prediction problem, analogously to how language generation architectures operate. We evaluate the system in multiple simulated trajectory scenarios, and show a significant performance increase of our model over baseline approaches. In addition, our realworld experiments with a robot arm show that users significantly prefer our natural language interface over traditional methods such as kinesthetic teaching or cost-function programming. Our study shows how the field of robotics can take advantage of large pre-trained language models towards creating more intuitive interfaces between robots and machines. Project webpage: https://arthurfenderbucker. github.io/NL_trajectory_reshaper/

show abstract

Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos

Cited by 7 publications

References 0 publications

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Learning Periodic Tasks from Human Demonstrations

Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers

Contact Info

Product

Resources

About