Computational neural mechanisms of goal-directed planning and problem solving

Abstract

The question of how animals and humans can solve arbitrary problems and achieve arbitrary goals remains open. Model-based and model-free reinforcement learning methods have addressed these problems, but they generally lack the ability to flexibly reassign reward value to various states as the reward structure of the environment changes. Research on cognitive control has generally focused on inhibition, rule-guided behavior, and performance monitoring, with relatively less focus on goal representations. From the engineering literature, control theory suggests a solution in that an animal can be seen as trying to minimize the difference between the actual and desired states of the world, and the Dijkstra algorithm further suggests a conceptual framework for moving a system toward a goal state. He we present a purely localist neural network model that can autonomously learn the structure of an environment and then achieve any arbitrary goal state in a changing environment without re-learning reward values. The model clarifies a number of issues inherent in biological constraints on such a system, including the essential role of oscillations in learning and performance. We demonstrate that the model can efficiently learn to solve arbitrary problems, including for example the Tower of Hanoi problem.

Foundations of human problem solving

“…After training, the model can generate novel action sequences to achieve arbitrary goal states. Adapted from (26).…”

“…Subjects were placed in one of four starting states and had to traverse one or two states to achieve a goal, by retrieving a key and subsequently using it to unlock a treasure chest for a reward ( Figure 2B). (26,29). B) Treasure Hunt task Both the GOLSA model and the human fMRI subjects performed a simple treasure hunt task, in which subjects were placed in one of four possible starting locations, then asked to generate actions to reach any of the other possible locations.…”

“…Here we explore a computational model that solves this more general problem of how the brain solves problems with changing goals (26), and we show how a number of brain regions may implement information processing in ways that correspond to specific model components. The model begins with a core premise: the brain constitutes a controltheoretic system, generating actions to minimize the discrepancy between actual and desired states.…”