Learning Reward Functions by Integrating Human Demonstrations and Preferences

Palan, Malayandi; Landolfi, Nicholas C.; Shevchuk, Gleb; Sadigh, Dorsa

doi:10.48550/arxiv.1906.08928

Cited by 12 publications

(18 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The concept of learning a hidden reward function from a user is widely used in various human-robot interaction frameworks, such as learning from demonstrations (LfD) [4], [17], learning from corrections [18], [19] and learning from preferences [1], [3], [12], [13], [17].…”

Section: A Related Workmentioning

confidence: 99%

Active Preference Learning using Maximum Regret

Wilde¹,

Kulić²,

Smith³

2020

Preprint

View full text Add to dashboard Cite

We study active preference learning as a framework for intuitively specifying the behaviour of autonomous robots. In active preference learning, a user chooses the preferred behaviour from a set of alternatives, from which the robot learns the user's preferences, modeled as a parameterized cost function. Previous approaches present users with alternatives that minimize the uncertainty over the parameters of the cost function. However, different parameters might lead to the same optimal behaviour; as a consequence the solution space is more structured than the parameter space. We exploit this by proposing a query selection that greedily reduces the maximum error ratio over the solution space. In simulations we demonstrate that the proposed approach outperforms other state of the art techniques in both learning efficiency and ease of queries for the user. Finally, we show that evaluating the learning based on the similarities of solutions instead of the similarities of weights allows for better predictions for different scenarios.

show abstract

Section: A Related Workmentioning

confidence: 99%

Active Preference Learning using Maximum Regret

Wilde¹,

Kulić²,

Smith³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Compared to (6), the new formulation in (7) has a new log term that is intentionally added to reflect humans' preference/desire. The new formulation allows us to learn a nonlinear objective function based on human preferences because human preferences among trajectory pairs can shape the loss towards the desired regions and away from the undesired regions.…”

Section: Training a Maximum Entropy Reinforcement Learning Algorithmmentioning

confidence: 99%

“…learning (see, e.g., [3], [4]) focuses on learning reward functions directly from human demonstrations. Human preferencebased learning (see, e.g., [5], [6], [7]) focuses on maximizing the volume removed from the distribution of the weight vector by asking a human to pick between trajectory pairs until reaching convergence. While both approaches provide important advances, they still lack much-needed data efficiency.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approach

Zhan

Tao

Cao

2020

Preprint

View full text Add to dashboard Cite

Human demonstrations can provide trustful samples to train reinforcement learning algorithms for robots to learn complex behaviors in real-world environments. However, obtaining sufficient demonstrations may be impractical because many behaviors are difficult for humans to demonstrate. A more practical approach is to replace human demonstrations by human queries, i.e., preference-based reinforcement learning. One key limitation of the existing algorithms is the need for a significant amount of human queries because a large number of labeled data is needed to train neural networks for the approximation of a continuous, high-dimensional reward function. To reduce and minimize the need for human queries, we propose a new GAN-assisted human preference-based reinforcement learning approach that uses a generative adversarial network (GAN) to actively learn human preferences and then replace the role of human in assigning preferences. The adversarial neural network is simple and only has a binary output, hence requiring much less human queries to train. Moreover, a maximum entropy based reinforcement learning algorithm is designed to shape the loss towards the desired regions or away from the undesired regions. To show the effectiveness of the proposed approach, we present some studies on complex robotic tasks without access to the environment reward in a typical MuJoCo robot locomotion environment. The obtained results show our method can achieve a reduction of about 99.8% human time without performance sacrifice.

show abstract

“…In this case, the good IRL methods need to have more expressive power and a more efficient framework. Research such as [9,15,21,30,41,50,54] was proposed to alleviate the above problems by using more expressive models like neural network and optimizing the input like ranking the demonstration in advance.…”

mentioning

confidence: 99%

Imitation Learning: Progress, Taxonomies and Opportunities

Zheng¹,

Verma²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Imitation learning aims to extract knowledge from human experts' demonstrations or artificially created agents in order to replicate their behaviours. Its success has been demonstrated in areas such as video games, autonomous driving, robotic simulations and object manipulation. However, this replicating process could be problematic, such as the performance is highly dependent on the demonstration quality, and most trained agents are limited to perform well in task-specific environments. In this survey, we provide a systematic review on imitation learning. We first introduce the background knowledge from development history and preliminaries, followed by presenting different taxonomies within Imitation Learning and key milestones of the field. We then detail challenges in learning strategies and present research opportunities with learning policy from suboptimal demonstration, voice instructions and other associated optimization schemes.

show abstract

Learning Reward Functions by Integrating Human Demonstrations and Preferences

Cited by 12 publications

References 0 publications

Active Preference Learning using Maximum Regret

Active Preference Learning using Maximum Regret

Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approach

Imitation Learning: Progress, Taxonomies and Opportunities

Contact Info

Product

Resources

About