Agency and goal-directed choice

Flexible Control as Surrogate Reward or Dynamic Reward Maximization

“…Consequently, while the difference in distance between token distributions across room options is slightly greater than that in the top panel, the difference in potentially obtained outcomes is lesser – if the preference for instrumental divergence is indeed rooted in an affordance of flexible control as subjective utilities change, the preference for the left over the right room in the bottom panel should be significantly less than that in the top panel. In contrast, if the preference scales with the information theoretic distance between token distributions, used in previous work to quantify agency in biological and artificial systems (9; 10), the preference for the left room should, if anything, be slightly greater in the bottom panel of Figure 1.…”

“…A rational explanation for the free-choice preference is that subjective outcome utilities often change from one moment to the next, and that free choice allows an agent to maximize long run rewards by switching between actions to flexibly produce whichever outcome is most preferred at any given time. Recently, it has been noted that free choice between actions that produce highly similar or identical outcomes affords no such flexibility and that, consequently, instrumental divergence – the degree to which action alternatives yield distinct outcomes – is an essential aspect of dynamic reward maximization (6; 7; 8). Here, the computational basis of a preference for instrumental divergence is probed, by contrasting the absolute and relative distances between outcome probability distributions.…”

“…However, recently, it has been noted that free choice between actions that produce highly similar or identical outcomes affords no such flexibility and that, consequently, instrumental divergence – the degree to which freely chosen actions yield distinct consequences – is an essential aspect of dynamic reward maximization (6; 7; 8; 9). Formally, the information theoretic (IT) distance between outcome probability distributions associated with action alternatives has been proposed to capture an agents control over its environment (6; 7; 8; 9; 10). Of course, without free choice, the IT distance between outcome distributions, although related to the predictability and diversity of outcomes, would not be instrumental and, consequently, would have no implications for flexible instrumental control.…”

“…Of course, without free choice, the IT distance between outcome distributions, although related to the predictability and diversity of outcomes, would not be instrumental and, consequently, would have no implications for flexible instrumental control. Greater instrumental divergence, thus, reflects both free choice and greater IT distance between outcome distributions (6; 7; 8; 9). Moreover, IT variables do not necessarily capture the full scope of instrumental flexibility – indeed, an explicit representation of instrumental divergence, however formalized, may not be necessary for adaptive choice.…”

“…Substantial evidence suggests that positive symptoms of schizophrenia (e.g., hallucinations and delusions) are associated with an exaggerated sense of agency (SoA) – defined by (11) as “the experience of controlling one’s own motor acts and, through them, the course of external events” – whether assessed using declarative statements of self- vs. external attribution (12; 13; 14), or implicitly, as altered time perception associated with voluntary actions (15) or interference by another’s movements on motor performance (16). Given the correspondence between instrumental divergence and contemporary accounts of agency, see (9) for review, this suggests that positive symptoms may also dysregulate a preference for instrumental divergence: Here, individual differences in dimensional positive schizotypy were used to predict the preference for greater instrumental divergence in neurotypical adults.…”