Where does value come from?

Abstract

The computational framework of reinforcement learning (RL) has allowed us to both understand biological brains and build successful artificial agents. However, in this article we highlight open challenges for RL as a model of animal behaviour in natural environments. We ask how the external reward function is designed for biological systems, and how we can account for the context sensitivity of valuation. We argue that rather than optimizing receipt of external reward signals, animals track current and desired internal states and seek to minimise the distance to goal across multiple value dimensions. Our framework can readily account for canonical phenomena observed in the fields of psychology, behavioural ecology, and economics, and recent findings from brain imaging studies of value-guided decision-making.

Homeostasis and soft robotics in the design of feeling machines

“…As a reviewer of Keramati and Gutkin's work put it, this makes "reinforcement learning accountable to homeostatic imperatives. " Recent work has extended this homeostatic RL framework to address high-level cognitive, social and economic behaviours 62 , and considered its relation to active inference 63 .…”

Foundations of human problem solving

“…Control theory has successfully accounted for the neural control of movement (20) but has not been applied to cognition more broadly. In this framework, a preferred decision prospect will define a set point, to be achieved by control-theoretic negative feedback controllers (21,22). Problem solving then requires 1) defining the goal state; 2) planning a sequence of state transitions to move the current state toward the goal; and 3) generating actions aimed at implementing the desired sequence of state transitions.…”

Computational neural mechanisms of goal-directed planning and problem solving

“…Agents behave so as to bring specific perceptual signals into the desired ranges (i.e., set points), such as the postural signals indicating a reach to a target location or the visual signals indicating that the agent has reached a desired location (Gibson, 1977;Hommel, Müsseler, Aschersleben, & Prinz, 2001). The concept has also been extended into value-based actions (Juechems & Summerfield, 2019), and is closely related to predictive coding (Friston, 2010) -at least in terms of minimizing the error between an actual and desired outcome, if not minimizing the error between the actual and predicted outcome (Ito, Stuphorn, Brown, & Schall, 2003).…”