Alterations in choice behavior by manipulations of world model

Abstract

How to compute initially unknown reward values makes up one of the key problems in reinforcement learning theory, with two basic approaches being used. Model-free algorithms rely on the accumulation of substantial amounts of experience to compute the value of actions, whereas in model-based learning, the agent seeks to learn the generative process for outcomes from which the value of actions can be predicted. Here we show that (i) "probability matching"-a consistent example of suboptimal choice behavior seen in humans -occurs in an optimal Bayesian model-based learner using a max decision rule that is initialized with ecologically plausible, but incorrect beliefs about the generative process for outcomes and (ii) human behavior can be strongly and predictably altered by the presence of cues suggestive of various generative processes, despite statistically identical outcome generation. These results suggest human decision making is rational and model based and not consistent with model-free learning.decision making | probability matching | reinforcement learning G iven a limited set of data about the world, what is the best thing to do? This question lies at the heart of all decision making, from simple everyday errands to elaborate and complex scientific experiments. If the reward amount for each possible action is known in advance, it is a straightforward process to make choices that maximize reward. In the real world, however, reward values are nearly always initially unknown and computing them is not trivial. Thus, understanding how to learn and compute reward is one of the key problems in reinforcement learning theory. Computing the optimal policy (i.e., determining the "best thing to do") requires acquiring one of two types of knowledge. In model-free learning, an agent must accumulate a substantial amount of experience regarding the consequences of taking various actions in various states, from which the average value of the states can be learned. In model-based learning, an agent must acquire a "world model," which constitutes beliefs about how the world generates outcomes in response to actions. Although both model-free and model-based reinforcement-learning algorithms have been the subject of much study in computer science and machine learning, model-free algorithms have been primarily used as models of human choice behavior.Whereas it is clear that our survival depends on the ability to make appropriate decisions from incomplete and ambiguous information, numerous studies in economics, psychology, and neuroscience have consistently found highly suboptimal behavior in seemingly simple decision tasks. Why is this? Consider the sequential binary decision task, which involves a choice between two options, one with a higher probability of success than the other (e.g., 70% vs. 30% of trials). The optimal strategy for this task is to determine which option has a higher probability of success and then choose only that option. Humans, however, tend to sample the alternatives in proportion to the options' respec...

The brain uses adaptive internal models of scene statistics for sensorimotor estimation and planning

“…This result suggests that n − 1 biases reflect subjects' estimates of the correlations in stimulus sequences but that subjects' estimates of stimulus correlations are positively biased. The positive-correlation bias observed in subjects' perceptual estimates of target speed is consistent with the literature on temporal dependencies in more cognitive binomial decision-making tasks (24)(25)(26). Kareev (24), for example, presented a sequence of binary items (Xs and Os) to subjects and asked them to predict the next item on each trial.…”

“…One proposal is that it reflects a strong prior on the statistical structure of the world, in which strong positive temporal correlations between events may be ubiquitous (26). The current study has shown that the brain can adapt its internal model of temporal correlations to more closely match the correlations of stimulus sequences in the proximal environment, although only partially on the short time scale used here (1 h).…”

“…In support of this idea, Green et al (26) have shown that subjects' betting behaviors on the outcomes of independent sequences of random binary events reflected an assumption of positive correlations when the outcomes were generated by a hidden random process in the environment. When subjects actively generated the outcomes themselves using motor behaviors that necessarily led to independence of outcomes, subjects correctly assumed independence.…”

A common rule for decision making in animal collectives across species

“…Pðx is goodÞ Pðx is goodÞ + Pðy is goodÞ : [2] This rule is known to be optimal when there is competition for resources (39,40) and when the estimated probabilities change in time (41)(42)(43)(44). Probability matching in Eq.…”

“…1. A common decision rule in animals, from insects to humans, is probability matching, according to which the probability of choosing a behavior is proportional to the estimated probability (35)(36)(37)(38)(39)(40)(41)(42)(43)(44),…”