Discovery of Hierarchical Representations for Efficient Planning

Abstract

<jats:title>Summary</jats:title><jats:p>We propose that humans spontaneously organize environments into clusters of states that support hierarchical planning, enabling them to tackle challenging problems by breaking them down into sub-problems at various levels of abstraction. People constantly rely on such hierarchical presentations to accomplish tasks big and small – from planning one’s day, to organizing a wedding, to getting a PhD – often succeeding on the very first attempt. We formalize a Bayesian model of hierarchy discovery that explains how humans discover such useful abstractions. Building on principles developed in structure learning and robotics, the model predicts that hierarchy discovery should be sensitive to the topological structure, reward distribution, and distribution of tasks in the environment. In five simulations, we show that the model accounts for previously reported effects of environment structure on planning behavior, such as detection of bottleneck states and transitions. We then test the novel predictions of the model in eight behavioral experiments, demonstrating how the distribution of tasks and rewards can influence planning behavior via the discovered hierarchy, sometimes facilitating and sometimes hindering performance. We find evidence that the hierarchy discovery process unfolds incrementally across trials. We also find that people use uncertainty to guide their learning in a way that is informative for hierarchy discovery. Finally, we propose how hierarchy discovery and hierarchical planning might be implemented in the brain. Together, these findings present an important advance in our understanding of how the brain might use Bayesian inference to discover and exploit the hidden hierarchical structure of the environment.</jats:p>

The reward-complexity trade-off in schizophrenia

“…Over the course of learning, the model adaptively compresses the policy so that it achieves the highest reward rate subject to a constraint on the average number of bits used to specify the policy. The implications of adaptive policy compression are wide-reaching: in addition to explaining quantitative aspects of choice perseveration (Gershman, 2020), it may also provide a normative explanation for different forms of action and state chunking observed experimentally (e.g., Dezfouli and Balleine, 2012, Tomov et al, 2020). Finally, the cost-sensitive actor-critic model suggests a computational rationale for the massive compression factor in the mapping from cortex to striatum (Bar-Gad et al, 2003).…”

What Is the Model in Model‐Based Planning?

“…This work highlights the fact that many sequential decision problems contain repeated sets of actions that are related to solving the same subgoals. At a high level, many of the computational approaches for learning which subgoals are useful for a given task involve identifying commonly traversed task states that serve as bottlenecks between large sets of similar states (Hengst, 2012; Tomov, Yagati, Kumar, Yang, & Gershman, 2020). For example, a doorway between two rooms represents a bottleneck linking any state in the first room to any state in the second room.…”

Generalization as diffusion: human function learning on graphs

“…When combined with the GP framework, it allows us to make Bayesian predictions about unobserved nodes. Even though previous work has investigated how people learn the relational structure of a graph (Kemp & Tenenbaum, 2008;Kemp, Tenenbaum, Griffiths, Yamada, & Ueda, 2006;Tomov, Yagati, Kumar, Yang, & Gershman, 2018), or infer properties of unobserved inputs (Kemp & Tenenbaum, 2009;Kemp, Shafto, & Tenenbaum, 2012), less is known about how people learn functions in discrete spaces with real-valued outputs.…”

“…Previous work has also investigated how people perform inference over graphs (Kemp & Tenenbaum, 2009, 2008Shafto, Kemp, Baraff, Coley, & Tenenbaum, 2005;Tomov et al, 2018). Whereas these studies were geared towards probing how people inferred underlying structure (Kemp & Tenenbaum, 2008) and how (implicit or explicit) represen-tations of structure influenced causal property judgments (Kemp & Tenenbaum, 2009;Shafto et al, 2005), the goal of our Subway Prediction Task was to study how people perform functional inference given explicit knowledge of a relational structure.…”