The Rostrolateral Prefrontal Cortex Mediates a Preference for High-Agency Environments

Abstract

The ability to exert flexible instrumental control over one's environment is a defining feature of adaptive decision-making. Here, we investigated neural substrates mediating a preference for environments with greater instrumental divergence, the distance between outcome probability distributions associated with alternative actions. A formal index of agency, instrumental divergence allows an organism to flexibly obtain the currently most desired outcome as preferences change. As such, it may have intrinsic utility, guiding decisions toward environments that maximize instrumental power. Consistent with this notion, we found that a measure of expected value that treats instrumental divergence as a reward surrogate provided a better account of male and female human participants' choice preferences than did a conventional model, sensitive only to monetary reward. Using model-based fMRI, we found that activity in the rostrolateral and ventromedial PFC, regions associated with abstract cognitive inferences and subjective value computations, respectively, scaled with the divergence-based account of expected value. Implications for a neural common currency of information theoretic and motivational variables are discussed.

Learning when effort matters: Neural dynamics underlying updating and adaptation to changes in performance efficacy

“…In particular, work in this area has shown that people are able to learn about and update their expectations of the likelihood that a given action will generate a given outcome (action-outcome contingency; Dickinson and Balleine 1995; Moscarello and Hartley 2017; Ly et al 2019). People preferentially, and more vigorously, select actions that reliably lead to desired outcomes (i.e., the more contingent those outcomes are on the action in question; Liljeholm et al 2011; Manohar et al 2017), and work in both animals (Balleine & O’Doherty, 2010) and humans (Norton & Liljeholm, 2020; Dorfman et al, 2021; Ligneul et al, 2022; Morris et al, 2022) has helped to characterize the neural systems that support this process of learning and action selection. However, given the focus on discrete actions and their immediate relationship with outcomes, research into these action-outcome contingencies is unable to capture key aspects that are unique to selection of control states.…”

Flexible Control as Surrogate Reward or Dynamic Reward Maximization

“…Whereas the difference between rooms in terms of outcome diversity and predictability is the same for self- and auto-play rooms, instrumental divergence is always zero in auto-play rooms. Importantly, the self- vs. auto-play manipulation also allows us to assess whether the frequently demonstrated preference for free-choice (6; 7; 8; 9) is modulated by the flexibility of control (i.e., instrumental divergence).…”

“…A rational explanation for the free-choice preference is that subjective outcome utilities often change from one moment to the next, and that free choice allows an agent to maximize long run rewards by switching between actions to flexibly produce whichever outcome is most preferred at any given time. Recently, it has been noted that free choice between actions that produce highly similar or identical outcomes affords no such flexibility and that, consequently, instrumental divergence – the degree to which action alternatives yield distinct outcomes – is an essential aspect of dynamic reward maximization (6; 7; 8). Here, the computational basis of a preference for instrumental divergence is probed, by contrasting the absolute and relative distances between outcome probability distributions.…”

“…However, recently, it has been noted that free choice between actions that produce highly similar or identical outcomes affords no such flexibility and that, consequently, instrumental divergence – the degree to which freely chosen actions yield distinct consequences – is an essential aspect of dynamic reward maximization (6; 7; 8; 9). Formally, the information theoretic (IT) distance between outcome probability distributions associated with action alternatives has been proposed to capture an agents control over its environment (6; 7; 8; 9; 10). Of course, without free choice, the IT distance between outcome distributions, although related to the predictability and diversity of outcomes, would not be instrumental and, consequently, would have no implications for flexible instrumental control.…”

“…Of course, without free choice, the IT distance between outcome distributions, although related to the predictability and diversity of outcomes, would not be instrumental and, consequently, would have no implications for flexible instrumental control. Greater instrumental divergence, thus, reflects both free choice and greater IT distance between outcome distributions (6; 7; 8; 9). Moreover, IT variables do not necessarily capture the full scope of instrumental flexibility – indeed, an explicit representation of instrumental divergence, however formalized, may not be necessary for adaptive choice.…”