Models of human preference for learning reward functions

Abstract

The utility of reinforcement learning is limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is to learn the reward function from human-generated preferences between pairs of trajectory segments. These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling preferences instead as arising from a different statistic: each segment's regret, a measure of a segment's deviation from optimal decision-making. Given infinitely many preferences generated according to regret, we prove that we can identify a reward function equivalent to the reward function that generated those preferences. We also prove that the previous partial return model lacks this identifiability property without preference noise that reveals rewards' relative proportions, and we empirically show that our proposed regret preference model outperforms it with finite training data in otherwise the same setting. Additionally, our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned. Overall, this work establishes that the choice of preference model is impactful, and our proposed regret preference model provides an improvement upon a core assumption of recent research.

Learning Optimal Advantage from Preferences and Mistaking It for Reward

“…[0, 1]) specifies the probability of an action given a state. Q ⇡ r and V ⇡ r refer respectively to the state-action value function and state value function for a 1 Appendix B.2 of Knox et al (2022) includes discounting. policy, ⇡, under r, and are defined as follows.…”

“…The right pair of segments instead illustrates the effect of differing start states, equivalent partial return, and the same end states (dark blue), and it permits an identical analysis. Knox et al (2022) showed that regret-based preferences have the desirable theoretical property of identifiability and that partial return does not. Further, the regret model better fit the dataset of human preferences they collected.…”

“…When learning from human preferences (in RLHF), the dominant model assumes that human preferences are determined only by each segment's accumulated reward, or partial return. Knox et al (2022) argued that the partial return preference model has fundamental flaws that are removed or ameliorated by instead assuming that human preferences are determined by the optimal advantage of each segment, which is a measure of deviation from optimal decision-making and is equivalent to the negated regret. This past work argues for the superiority of the regret preference model (1) by intuition, regarding how humans are likely to give preferences (e.g., see Fig.…”

Eliciting User Preferences for Personalized Multi-Objective Decision Making through Comparative Feedback

“…These works typically assume that noisy human preferences over a pair of trajectories are correlated with the difference in their utilities (i.e., the reward acts as a latent term predictive of preference). Many contemporary methods estimate the latent reward by minimizing the cross-entropy loss between the reward-based predictions and the human-provided preferences (i.e., finding the reward that maximizes the likelihood of the observed preferences) [16,20,27,26,33].…”

Transformers are Adaptable Task Planners

“…[24] propose VAE to learn user preferences for spatial arrangement based on just the final state, while our approach models temporal preference from demonstrations. Preference-based RL learns rewards based on human preferences [57,58,59,60], but do not generalize to unseen preferences. On complex long-horizon tasks, modeling human preferences enables faster learning than RL, even with carefully designed rewards [61].…”