Hidden Reward: Affect and Its Prediction Errors as Windows Into Subjective Value

Abstract

Scientists increasingly apply concepts from reinforcement learning to affect, but which concepts should apply? And what can their application reveal that we cannot know from directly observable states? An important reinforcement learning concept is the difference between reward expectations and outcomes. Such reward prediction errors have become foundational to research on adaptive behavior in humans, animals, and machines. Owing to historical focus on animal models and observable reward (e.g., food or money), however, relatively little attention has been paid to the fact that humans can additionally report correspondingly expected and experienced affect (e.g., feelings). Reflecting a broader “rise of affectivism,” attention has started to shift, revealing explanatory power of expected and experienced feelings—including prediction errors—above and beyond observable reward. We propose that applying concepts from reinforcement learning to affect holds promise for elucidating subjective value. Simultaneously, we urge scientists to test—rather than inherit—concepts that may not apply directly.

Affective prediction errors in persistence and escalation of aggression.

“…Affective PEs are not merely an affective mirror image of prediction errors as they are usually investigated (see Vollberg & Sander, 2024). While we have provided a potentially generative framework to study behavior in domains like aggression, it is important to reemphasize the differences between affective and outcome prediction errors.…”

“…From an affective forecasting perspective, however, affective traces are quick to fade and expectations may not even be relevant at all for how we feel and act after we experience an outcome (Golub et al, 2009). In principle, because affect (as opposed to outcomes in the environment) originates in the experiencer, prediction errors about affect may follow different rules than prediction errors about outcomes, despite their superficial similarities (Vollberg & Sander, 2024). As we test our novel framework in the context of aggression, we will thus make sure to keep these alternatives in mind empirically.…”

“…A recent study sought to address this gap in the context of economic games, separating emotion prediction errors (measuring both valence and arousal) and outcome prediction errors, suggesting that they might independently and systematically contribute to choice behavior (Heffner et al, 2021). That there is explanatory power of affective PEs above and beyond outcome prediction errors has at least two implications that complement traditional reward learning frameworks (see Vollberg & Sander, 2024 for a detailed discussion of how affective PEs relate to traditional reward learning frameworks). First, verbally reported affect can be used to approximate subjective value that reflects variations in an organism’s internal and external states (Juechems & Summerfield, 2019).…”

The added value of affective processes for models of human cognition and learning

“…As such, we suggest that considering affective processes is pivotal to the modeling of human cognition, and especially of learning. Affective processes are indeed central to-and exert a pervasive influence on-how humans learn (e.g., Öhman & Mineka, 2001;Vollberg & Sander, 2024;Wuensch et al, 2021). Below, we illustrate how emotion and other affective phenomena are central to human learning across various domains, with a particular focus on reinforcement learning, knowledge acquisition, and social learning.…”