Undermatching is a consequence of policy compression

Abstract

The matching law describes the tendency of agents to match the ratio of choices allocated to the ratio of rewards received when choosing among multiple options (Herrnstein, 1961). Perfect matching, however, is infrequently observed. Instead, agents tend to undermatch, or bias choices towards the poorer option. Overmatching, or the tendency to bias choices towards the richer option, is rarely observed. Despite the ubiquity of undermatching, it has received an inadequate normative justification. Here, we assume agents not only seek to maximize reward, but also seek to minimize cognitive cost, which we formalize as policy complexity (the mutual information between actions and states of the environment). Policy complexity measures the extent to which an agent’s policy is state-dependent. Our theory states that capacity-constrained agents (i.e., agents that must compress their policies to reduce complexity), can only undermatch or perfectly match, but not overmatch, consistent with the empirical evidence. Moreover, we validate a novel prediction about which task conditions exaggerate undermatching. Finally, we argue that a reduction in undermatching with higher dopamine levels in patients with Parkinson’s disease is consistent with an increased policy complexity.Significance statementThe matching law describes the tendency of agents to match the ratio of choices allocated to different options to the ratio of reward received. For example, if option A yields twice as much reward as option B, matching states that agents will choose option A twice as much. However, agents typically undermatch: they choose the poorer option more frequently than expected. Here, we assume that agents seek to simultaneously maximize reward and minimize the complexity of their action policies. We show that this theory explains when and why undermatching occurs. Neurally, we show that policy complexity, and by extension undermatching, is controlled by tonic dopamine, consistent with other evidence that dopamine plays an important role in cognitive resource allocation.

Bayesian Reinforcement Learning With Limited Cognitive Load

“…There is a long, rich literature exploring the natural limitations on time, knowledge, and cognitive capacity faced by human (and animal) decision makers (Amir et al, 2020 ; Bhui et al, 2021 ; Binz & Schulz, 2022 ; Brown et al, 2022 ; Gershman et al, 2015 ; Gigerenzer & Goldstein, 1996 ; Griffiths et al, 2015 ; Ho et al, 2022 ; Icard & Goodman, 2015 ; Lieder & Griffiths, 2020 ; Newell & Simon, 1972 ; Newell et al, 1958 ; Prystawski et al, 2022 ; Simon, 1956 , 1982 ; Shugan, 1980 ; Vul et al, 2014 ). Crucially, our focus is on a recurring theme throughout this literature of modeling these limitations on cognitive capabilities as being information-theoretic in nature (Bari & Gershman, 2022 ; Botvinick et al, 2015 ; Gershman, 2020 , 2023 ; Gershman & Lai, 2020 ; Ho et al, 2020 ; Jakob & Gershman, 2022 ; Lai & Gershman, 2021 ; Mikhael et al, 2021 ; Parush et al, 2011 ; Peng, 2005 ; Sims, 2003 , 2016 , 2018 ; Zénon et al, 2019 ).…”

Mechanisms of adjustments to different types of uncertainty in the reward environment across mice and monkeys

“…Functionally, this algorithm implements the identical baiting mechanism used in the mouse experiment (also see Eq. 2 in Bari & Gershman, 2022). The baseline reward probabilities ( P 0 ) were chosen from 20 different sets of reward schedules ( Figure S1D ).…”

“…All surgical and experimental procedures were in accordance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals and approved by the Johns Hopkins University Animal Care and Use Committee. This experimental setup and some analyses of the data for electrophysiological recordings and matching behavior have been reported previously (Bari et al, 2019; Trepka et al, 2021; Bari & Gershman, 2022). All analyses reported in this study are novel.…”

“…Functionally, this algorithm implements the identical baiting mechanism used in the mouse experiment (also see Eq. 2 in Bari & Gershman, 2022).…”