Dopamine Reward Prediction Error Responses Reflect Marginal Utility

Abstract

SummaryBackgroundOptimal choices require an accurate neuronal representation of economic value. In economics, utility functions are mathematical representations of subjective value that can be constructed from choices under risk. Utility usually exhibits a nonlinear relationship to physical reward value that corresponds to risk attitudes and reflects the increasing or decreasing marginal utility obtained with each additional unit of reward. Accordingly, neuronal reward responses coding utility should robustly reflect this nonlinearity.ResultsIn two monkeys, we measured utility as a function of physical reward value from meaningful choices under risk (that adhered to first- and second-order stochastic dominance). The resulting nonlinear utility functions predicted the certainty equivalents for new gambles, indicating that the functions’ shapes were meaningful. The monkeys were risk seeking (convex utility function) for low reward and risk avoiding (concave utility function) with higher amounts. Critically, the dopamine prediction error responses at the time of reward itself reflected the nonlinear utility functions measured at the time of choices. In particular, the reward response magnitude depended on the first derivative of the utility function and thus reflected the marginal utility. Furthermore, dopamine responses recorded outside of the task reflected the marginal utility of unpredicted reward. Accordingly, these responses were sufficient to train reinforcement learning models to predict the behaviorally defined expected utility of gambles.ConclusionsThese data suggest a neuronal manifestation of marginal utility in dopamine neurons and indicate a common neuronal basis for fundamental explanatory constructs in animal learning theory (prediction error) and economic decision theory (marginal utility).

Models of Affective Decision-Making: How Do Feelings Predict Choice?

“…The feeling function that best related feelings to value was revealed to be concave for gains and convex for losses, much as the value function in prospect theory (Kahneman & Tversky, 1979;Tversky & Kahneman, 1992) and other nonlinear utility functions (Bernoulli, 1954;Fox & Poldrack, 2014;Stauffer, Lak, & Schultz, 2014 The right column shows correlations between the propensity to gamble and loss-gain weight difference for expected feelings (top) and experienced feelings (bottom). Best-fitting regression lines are shown for each set of data.…”

Active Inference: A Process Theory

“…This endorses the notion of reward prediction errors as an update signal that the brain might use, in the sense that if posterior beliefs based on current observations reduce the expected free energy, relative to prior beliefs, then precision will increase . This can be related to dopamine discharges that have been interpreted in terms of changes in expected reward (Schultz & Dickinson, 2000;Fiorillo et al, 2003) and marginal utility (Stauffer, Lak, & Schultz, 2014). We have previously considered the intimate (monotonic) relationship between expected precision and expected utility in this context (see Friston et al, 2014, for a fuller discussion).…”

For Value or for Worth? Part 2: A Neuroeconomical Thought-Action-Mood-Space Modulated by (Un-)Certainty as Sign- or Goal-tracking

“…The need for comparison of rewards, often without a "common quality", calls for a ranking on a "common scale of values", on which the encoding of the RPE as "subjective value" is proposed as a, questionably, "ideal way" of steering economic decisions [31]. This "value" in reality though, as we can see, is a variable composite of worth (utility) and, at times incentive, value (effort), which is hidden behind of-ten dichotomic decisions.…”

“…This "value" in reality though, as we can see, is a variable composite of worth (utility) and, at times incentive, value (effort), which is hidden behind of-ten dichotomic decisions. The variation of rewards in only one attribute is said not to allow the "isolation" of subjective preference -as far as sooner, more certain, and more should be preferred as better [31]. But beyond this we want what we like at discounted objective and subjective effort, thus the RPE changes to the residual RPE(-∆%Effme).…”

“…This worth divided by recent effort answers to the question "Was it worth the effort?" -risking regret [31] and updates the net RPE [32].…”