Habits without Values

Abstract

Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as behaviors arising from model-free reinforcement learning (RL) mechanisms, which typically represent the expected value associated with the possible outcomes of each action before one of those actions is chosen. Traditionally, however, habits have been understood as arising from mechanisms that are independent of outcomes. Here, we develop a computational model instantiating this traditional view, in which habits are acquired through the direct strengthening of recently taken actions, independent of outcome. We demonstrate how this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the dependence of formation rates on the reinforcement schedule. The model also explains the prevalent observation of perseveration in repeated choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors, and help to better guide research into the neural mechanisms underlying control of instrumental behaviors more generally.

Studying Human Habits in Societal Context: Examining Support for a Basic Stimulus–Response Mechanism

“…In such a case, taking the car and driving to work is chunked and routinized up to the level of being directly controlled by bottom-up processes that receive no feedback from goal states. Although this is conceivable, we do not know yet whether such a form of habit learning is possible (but see Miller et al, 2019, for a Hebbian account of habit formation).…”

Computational approaches to habits in a model-free world

“…A second modification offers an opportunity for a unification of use-dependent learning [4,6] and MF-RL models of habits. Assuming that synaptic plasticity is the result of Hebbian learning mechanisms ubiquitous throughout the brain [29], RPEs are thought to be modulating the rate of synaptic plasticity [30 ,31].…”

“…Arbitration models differ in their assumption about the criteria by which an arbitrator weights the contribution of each system (e.g. the respective uncertainty or expected inaccuracy [13], or reliability of the two systems [41], based on cost-benefit analyses [42,43], or based on deviations of the reward rate from the expected reward rate [6]). Because of the significant computational effort associated with evaluating the performance and predictions of constituent models during arbitration, these models do not speak to the benefits of hierarchical integration of behavior.…”

“…A second category of models focuses on use-dependent learning [4]. These models predict that habitual behaviors evolve merely from extended context-dependent repetition of a behavior [5,6].…”

Generating options and choosing between them rely on distinct forms of value representation

“…In Study 3 we explored the relative influence of two factors intimately associated with habitual control: historical value (Dolan & Dayan, 2013) and historical choice frequency (Miller, Shenhav, & Ludvig, 2019). Naturally, these are often correlated because we tend to choose the things we value most.…”