Human irrationality: both bad and good for reward inference

Abstract

Assuming humans are (approximately) rational enables robots to infer reward functions by observing human behavior. But people exhibit a wide array of irrationalities, and our goal with this work is to better understand the effect they can have on reward inference. The challenge with studying this effect is that there are many types of irrationality, with varying degrees of mathematical formalization. We thus operationalize irrationality in the language of MDPs, by altering the Bellman optimality equation, and use this framework to study how these alterations would affect inference.We find that wrongly modeling a systematically irrational human as noisy-rational performs a lot worse than correctly capturing these biases -so much so that it can be better to skip inference altogether and stick to the prior! More importantly, we show that an irrational human, when correctly modelled, can communicate more information about the reward than a perfectly rational human can. That is, if a robot has the correct model of a human's irrationality, it can make an even stronger inference than it ever could if the human were rational. Irrationality fundamentally helps rather than hinder reward inference, but it needs to be correctly accounted for. We explain this theoretically by showing that irrationality can increase the mutual information between the policy and the reward. We also replicate the result in an autonomous driving domain with myopic/greedy demonstrations.

Utility is in the Eye of the User: A Critique of NLP Leaderboards

“…Prospect theory, despite being highly influential in behav-ioral economics, has had a fairly muted impact in machine learning, with work concentrated in human-robot interaction (Kwon et al, 2020;Sun et al, 2019;Chan et al, 2021). Learning from sparse binary feedback is a staple of information retrieval and recommender systems (He et al, 2017;Koren et al, 2009), although to our knowledge it has not been used to generate open-ended text.…”

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

“…We study several different models of human biases. Following (Chan, Critch, and Dragan 2021), we formalize each bias as a particular modification to the standard Bellman update, resulting in a modified value function which we use to determine the resulting policy and simulate choices from a biased human. We assume that the person is Boltzmann rational under their biased value function.…”

Zero-Shot Assistance in Sequential Decision Problems

“…A subset of this literature has looked specifically into the feasibility and utility of learning both agents' reward functions and biases (Evans and Goodman 2015;Evans, Stuhlmueller, and Goodman 2016). Chan, Critch, and Dragan (2021) found that biases can make agents' behaviour more informative of their reward function, and that incorrectly modeling biases can result in poor reward inference. Armstrong and Mindermann (2018), however, show that jointly identifying biases and reward from observations is not always possible.…”