Multi-step planning in the brain

Abstract

Decisions in the natural world are rarely made in isolation. Each action that an organism selects will affect the future situations in which it finds itself, and those situations will in turn affect the future actions that are available. Achieving real-world goals often requires successfully navigating a sequence of many actions. An efficient and flexible way to achieve such goals is to construct an internal model of the environment, and use it to plan behavior multiple steps into the future. This process is known as multi-step planning, and its neural mechanisms are only beginning to be understood. Here, we review recent advances in our understanding of these mechanisms, many of which take advantage of multi-step decision tasks for humans and animals. Multi-Step Planning Shares Neural Mechanisms with Associative Learning Work on the neural mechanisms of planning builds upon a rich body of work investigating cognitive capacities upon which planning may depend. One of these is the ability to associate stimuli or actions with specific expected outcomes (e.g. walk south → arrive at lab; walk north → arrive at coffee shop). Behavior guided by such outcome-specific associations can be thought of as exercising a simple one-step form of planning. Researchers have developed several selective assays of this capacity and have used them extensively to identify and characterize the neural structures that support outcome-specific associations [23]. A second cognitive capacity necessary for multi-step planning is "inference": the ability to combine separately-learned associations in order to form new associations between items that may never have been encountered together before (e.g. walk north → arrive at coffee shop; place order → receive coffee; ∴ to obtain coffee, walk north). Researchers have developed assays of this capacity as well, and have used them to identify neural structures necessary for combining separately-learned associations [e.g. 24,25,26]. Mechanisms for outcome-specific associations and for inference are universally present in theoretical accounts of multi-step planning (see box). Recent work has borne out the idea that the neural structures which support outcome-specific associations and inference in associative learning tasks likely support planning in multi-step decision tasks as well. Orbitofrontal Cortex and Model-Based Prediction Regions of frontal cortex belonging to the orbital network (especially areas LO and AIv in rodents and areas 13 and 11l in primates, often referred to as "OFC" or "lateral OFC") play an important role both in outcome-specific associations and in inference. Lesions or inactivations of OFC have been shown to impair performance on behavioral assays of these capacities in rodents [27-29] and to impair use of outcome-specific associations in nonhuman primates [30]. Recent data have extended these findings to humans as well, showing that disruption of OFC activity impairs performance on assays of outcome-specific associations and of inference [31,32]. Correspondingly, neural acti...

The Best Laid Plans: Computational Principles of Anterior Cingulate Cortex

“…Growing evidence indicates that ACC, together with other brain areas [20], contributes to MB-RL. For example, a prominent computational model of ACC contains units that exhaustively predict all possible states of the task environment [21] (see also [22]).…”

Replay in minds and machines

“…During behavior, replay events can switch between reflecting immediately preceding, upcoming or more remote episodes, depending on the behavioral state of the animal at the time of replay (Pfeiffer and Foster, 2013;Ólafsdóttir et al, 2017). Even single replay events can depict more than one trajectory, such as the next one and the path the animal will take after reaching the goal location (Pfeiffer and Foster, 2013), as if representing a multi-step planning process (Foster, 2017;Miller and Venditto, 2021).…”

Flexible Path Planning Through Vicarious Trial and Error

“…A hallmark of cognitive behavior is flexible planning [Miller and Venditto, 2021]. Although the present study and many others concentrate on navigation when investigating flexible planning, such spatial behavior can be extended to non-spatial behavior.…”

“…Flexible planning is an important aspect of cognition that is especially useful when achieving a goal under uncertain conditions. Multi-step planning can be thought of as a process that uses a cognitive map to guide a sequence of actions towards a goal [Miller and Venditto, 2021]. In the context of spatial navigation, humans and other animals have the ability to choose alternate routes when necessary.…”