Choice history effects describe how future choices depend on the history of past choices. In experimental tasks this is typically framed as a bias because it often diminishes the experienced reward rates. However, in natural habitats, choices made in the past constrain choices that can be made in the future. For foraging animals, the probability of earning a reward in a given patch depends on the degree to which the animals have exploited the patch in the past. One problem with many experimental tasks that show choice history effects is that such tasks artificially decouple choice history from its consequences on reward availability over time. To circumvent this, we use a variable interval (VI) reward schedule that reinstates a more natural contingency between past choices and future reward availability. By examining the behavior of optimal agents in the VI task we discover that choice history effects observed in animals serve to maximize reward harvesting efficiency. We further distil the function of choice history effects by manipulating first- and second-order statistics of the environment. We find that choice history effects primarily reflect the growth rate of the reward probability of the unchosen option, whereas reward history effects primarily reflect environmental volatility. Based on observed choice history effects in animals, we develop a reinforcement learning model that explicitly incorporates choice history over multiple time scales into the decision process, and we assess its predictive adequacy in accounting for the associated behavior. We show that this new variant, known as the double trace model, has a higher performance in predicting choice data, and shows near optimal reward harvesting efficiency in simulated environments. These results suggests that choice history effects may be adaptive for natural contingencies between consumption and reward availability. This concept lends credence to a normative account of choice history effects that extends beyond its description as a bias.

Choice history effects describe how future choices depend on the history of past choices. Choice history effects are typically framed as a bias rather than an adaptive phenomenon because the phenomenon generally degrades reward rates in experimental tasks. How-ever, in natural habitats, choices made in the past constrain choices that can be made in the future. For foraging animals, the probability of obtaining a reward in a given patch depends on the degree to which the animals have exploited the patch in the past. One problem with many experimental tasks that show choice history effects is that such tasks artificially decouple choice history from its consequences in regard to reward availability over time. To circumvent this, we used a variable interval (VI) reward schedule that reinstates a more natural contingency between past choices and future reward availability. By manipulating first- and second-order statistics of the environment, we dissociated choice history, reward history, and reaction times. We found that choice history effects reflect the growth rate of the reward probability of the unchosen option, reward history effects reflect environmental volatility, and reaction time reflects overall reward rate. By testing in mice and humans, we show that the same choice history effects can be generalized across species and that these effects are similar to those observed in optimal agents. Furthermore, we develop a new reinforcement learning model that explicitly incorporates choice history over multiple timescales into the decision process, and we examine its predictive adequacy in accounting for the associated behavioral data. We show that this new variant, known as the double trace model, has a higher predictive adequacy of choice data, in addition to better reward harvesting efficiency in simulated environments. Finally, we show that the choice history effects emerge in optimal models of foraging in habitats with diminishing returns, thus linking this phenomenon to a wider class of optimality models in behavioral ecology. These results suggests that choice history effects may be adaptive for natural contingencies between consumption and reward availability. This concept lends credence to a normative account of choice history effects that extends beyond its description as a bias.

When foraging for food, animals must track the sensory events of their environment and their own actions over time. Memory of these sensorimotor events is crucial for learning the values of different options and foraging policies 1. To investigate the role of the medial prefrontal cortex (mPFC) in foraging behavior, we conducted experiments on mice using foraging tasks that required integration of past oro-sensory rewards and past choices. We found that the mPFC selectively represents sensorimotor events, which organize into a spatiotemporal map encoding location and temporal delay of past rewards and choices relative to the animal's current epoch in time. These representations of sensorimotor events, which we refer to as sensorimotor state representations, play a critical role in foraging behavior. Inactivation of the mPFC affected the integration of past rewards and choices into the mice's decisions, leading to a decrease in reward harvesting efficiency, particularly for longer temporal delays. Behavioral models that compute values and policy failed to capture the representations in mPFC. Our results suggest that the mPFC plays a critical role in representing sensorimotor states independently of value and policy computations. This highlights the importance of considering sensorimotor state representation in the mPFC in understanding foraging behavior.

No abstract