Deep reinforcement learning from human preferences

Christiano, Paul; Leike, Jan; Brown, T. B.; Martic, Miljan; Legg, Shane; Amodei, Dario

doi:10.48550/arxiv.1706.03741

Cited by 76 publications

(79 citation statements)

References 14 publications

(23 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PREFERENCES: A reward model learned using preference comparisons, similar to the approach from Christiano et al (2017). We synthesize preference labels based on ground-truth return.…”

Section: Methodsmentioning

confidence: 99%

“…Rewards can be learned from supervision provided by explicit labels. For example, human-labeled preferences between demonstrations can be used to learn reward functions (Christiano et al, 2017). Rewards can also be labeled on a per-timestep basis, allowing for learning via supervised regression (Cabi et al, 2019).…”

Section: Learning Reward Functionsmentioning

confidence: 99%

“…Tasks such as game playing have simple, natural reward functions that enable learning of effective policies (Tesauro, 1995;Mnih et al, 2013;Silver et al, 2017); however, many tasks, such as those involving human interaction, multiple competing objectives, or high-dimensional state spaces, do not have easy to define reward functions (Knox et al, 2021;Holland et al, 2013;Cabi et al, 2019). In these cases, reward functions can be learned from demonstrations, explicit preferences, or other forms of reward supervision (Ziebart et al, 2008;Christiano et al, 2017;Cabi et al, 2019). However, evaluating the quality of learned reward functions, which may be too complicated to manually inspect, remains an open challenge.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dynamics-Aware Comparison of Learned Reward Functions

Wulfe¹,

Balakrishna²,

Ellis³

et al. 2022

Preprint

View full text Add to dashboard Cite

The ability to learn reward functions plays an important role in enabling the deployment of intelligent agents in the real world. However, comparing reward functions, for example as a means of evaluating reward learning methods, presents a challenge. Reward functions are typically compared by considering the behavior of optimized policies, but this approach conflates deficiencies in the reward function with those of the policy search algorithm used to optimize it. To address this challenge, Gleave et al. (2020) propose the Equivalent-Policy Invariant Comparison (EPIC) distance. EPIC avoids policy optimization, but in doing so requires computing reward values at transitions that may be impossible under the system dynamics. This is problematic for learned reward functions because it entails evaluating them outside of their training distribution, resulting in inaccurate reward values that we show can render EPIC ineffective at comparing rewards. To address this problem, we propose the Dynamics-Aware Reward Distance (DARD), a new reward pseudometric. DARD uses an approximate transition model of the environment to transform reward functions into a form that allows for comparisons that are invariant to reward shaping while only evaluating reward functions on transitions close to their training distribution. Experiments in simulated physical domains demonstrate that DARD enables reliable reward comparisons without policy optimization and is significantly more predictive than baseline methods of downstream policy performance when dealing with learned reward functions. †

show abstract

“…PREFERENCES: A reward model learned using preference comparisons, similar to the approach from Christiano et al (2017). We synthesize preference labels based on ground-truth return.…”

Section: Methodsmentioning

confidence: 99%

Section: Learning Reward Functionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dynamics-Aware Comparison of Learned Reward Functions

Wulfe¹,

Balakrishna²,

Ellis³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Apart from the vast array of work from the Imitation Learning literature (Zheng et al 2021), which is largely motivated by the difficulty of designing reward functions and seeks to learn a policy from expert demonstrations instead, several other approaches have been specifically aimed at the reward specification problem. Some of these approaches introduce a human in the loop to either guide the agent towards the desired behavior (Christiano et al 2017) or to prevent it from making catastrophic errors while exploring the environment (Saunders et al 2017). While our approach of using CMDPs for behavior specification also seeks to make better use of human knowledge, we focus on tasks where the human input can be provided as a zero-shot interaction ahead of training time by simply specifying indicator cost functions and their corresponding thresholds rather than requiring human feedback during the training process.…”

Section: Reward Specificationmentioning

confidence: 99%

Direct Behavior Specification via Constrained Reinforcement Learning

Roy¹,

Girgis²,

Romoff³

et al. 2021

Preprint

View full text Add to dashboard Cite

The standard formulation of Reinforcement Learning lacks a practical way of specifying what are admissible and forbidden behaviors. Most often, practitioners go about the task of behavior specification by manually engineering the reward function, a counter-intuitive process that requires several iterations and is prone to reward hacking by the agent. In this work, we argue that constrained RL, which has almost exclusively been used for safe RL, also has the potential to significantly reduce the amount of work spent for reward specification in applied Reinforcement Learning projects. To this end, we propose to specify behavioral preferences in the CMDP framework and to use Lagrangian methods, which seek to solve a min-max problem between the agent's policy and the Lagrangian multipliers, to automatically weigh each of the behavioral constraints. Specifically, we investigate how CMDPs can be adapted in order to solve goal-based tasks while adhering to a set of behavioral constraints and propose modifications to the SAC-Lagrangian algorithm to handle the challenging case of several constraints. We evaluate this framework on a set of continuous control tasks relevant to the application of Reinforcement Learning for NPC design in video games.

show abstract

“…E.g., until now, many open-sourced RL datasets [19,22,27] are shared as a set of state/action pairs comparable to example/label pairs in supervised learning. Although this format is convenient for some algorithms [36], it discards the temporal information initially present in the data and prevents building methods exploiting such information [34,24,25,9,14]. Even when temporal information is present in a dataset, what constitutes a step, a transition, and an episode is not always consistent.…”

Section: Introductionmentioning

confidence: 99%

RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

Ramos¹,

Girgin²,

Hussenot³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce RLDS (Reinforcement Learning Datasets), an ecosystem for recording, replaying, manipulating, annotating and sharing data in the context of Sequential Decision Making (SDM) including Reinforcement Learning (RL), Learning from Demonstrations, Offline RL or Imitation Learning. RLDS enables not only reproducibility of existing research and easy generation of new datasets, but also accelerates novel research. By providing a standard and lossless format of datasets it enables to quickly test new algorithms on a wider range of tasks. The RLDS ecosystem makes it easy to share datasets without any loss of information and to be agnostic to the underlying original format when applying various data processing pipelines to large collections of datasets. Besides, RLDS provides tools for collecting data generated by either synthetic agents or humans, as well as for inspecting and manipulating the collected data. Ultimately, integration with TFDS [3] facilitates the sharing of RL datasets with the research community.

show abstract

Deep reinforcement learning from human preferences

Cited by 76 publications

References 14 publications

Dynamics-Aware Comparison of Learned Reward Functions

Dynamics-Aware Comparison of Learned Reward Functions

Direct Behavior Specification via Constrained Reinforcement Learning

RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

Contact Info

Product

Resources

About