Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for flexible reinforcement learning

Abstract

A hallmark of human intelligence is our ability to compositionally generalise: that is, to recompose familiar knowledge components in novel ways to solve new problems. For instance, a talented musician can conceivably transfer her knowledge of flute fingerings and guitar songs to play guitar music on a piccolo for the first time. Yet there are also instances where it can be helpful to learn and transfer not just individual task components, but entire structures or substructures, particularly whenever these recur in natural tasks (e.g., in bluegrass music one might transfer the joint structure of finger movements and musical scales from one stringed instrument to another). Prior theoretical work has explored how agents can learn and generalize task components or entire latent structures, but a satisfactory account for how a single agent can simultaneously satisfy the two competing demands is still lacking. Here, we propose a hierarchical model-based agent that learns and transfers individual task components as well as entire structures by inferring both through a non-parametric Bayesian model of the task. It maintains a factorised representation of task components through a hierarchical Dirichlet process, but it also represents different possible covariances between these components through a standard Dirichlet process. We validate our approach on a variety of navigation tasks covering a wide range of statistical correlations between task components and show that this hierarchical framework can also be applied to improve generalisation and transfer in hierarchical tasks with goal/subgoal structures.

Schemas, reinforcement learning, and the medial prefrontal cortex

“…For example, 'adding salt' can be a subtask that starts upon tasting bland food, continues with a policy that includes reaching for the salt shaker, grasping it, and shaking it over the food, and ends when the subgoal is reached: there is salt on the food. Subtasks can be used across tasks 89,96,97 (e.g., 'adding salt' is used by 'dining at a restaurant' and 'eating at home'). The term "subgoal" distinguishes the termination state of the subtask (food is salted) from the overall goal of the task (having a full stomach).…”

Learning to weigh competing moral motivations.

“…As we increasingly confront a bombardment of moral issues on the national and international stage, from how we deal with an emerging climate crisis to social justice issues such as racial reparations, it has never been more important to understand the ways in which human moral judgments systematically emerge and are updated from the social inputs around us. Although social computational models of learning are still in their infancy, formal models of learning are becoming increasingly sophisticated in their ability to capture the building blocks of human learning and reasoning that are relevant to the moral domain, such as concept abstraction (Gershman et al, 2015;R. G. Liu & Frank, 2021;Tenenbaum et al, 2011) and mental simulation (Ho et al, 2022).…”