Generalizing to generalize: when (and when not) to be compositional in task structure learning

Abstract

Humans routinely face novel environments in which they have to generalize in order toact adaptively. However, doing so involves the non-trivial challenge of deciding which aspects of a task domain to generalize. While it is sometimes appropriate to simply re-use a learned behavior, often adaptive generalization entails recombining distinct components of knowledge acquired across multiple contexts. Theoretical work has suggested a computational trade-off in which it can be more or less useful to learn and generalize aspects of task structure jointly or compositionally, depending on previous task statistics, but empirical studies are lacking. Here we develop a series of navigation tasks which manipulate the statistics of goal values (“what to do”) and state transitions (“how to do it”) across contexts, and assess whether human subjects generalize these task components separately or conjunctively. We find that human generalization is sensitive to the statistics of the previously experienced task domain, favoring compositional or conjunctive generalization when the task statistics are indicative of such structures, and a mixture of the two when they are more ambiguous. These results support the predictions of a normative “meta-generalization learning” agent that does not only generalize previous knowledge but also generalizes the statistical structure most likely to support generalization.Author NoteThis work was supported in part by the National Science Foundation Proposal 1460604 “How Prefrontal Cortex Augments Reinforcement Learning” to MJF. We thank Mark Ho for providing code used in the behavioral task. We thank Matt Nassar for helpful discussions. Correspondence should be addressed to Nicholas T. Franklin (nfranklin@fas.harvard.edu) or Michael J. Frank (michael_frank@brown.edu).

Reward-predictive representations generalize across tasks in reinforcement learning

“…When a musician picks up a banjo, they may quickly recognize its similarity to other string instruments—even those with alternate tuning—and efficiently learn to play a scale; the same musician may re-use a different structure when attempting to master the accordion. Previous theoretical work relied on non-parametric Bayesian clustering models that assess which of several previously seen structures might apply to a novel situation and be flexibly combined in a compositional fashion [ 3 ], a strategy supported by empirical studies in humans [ 16 ]. However, such an approach still requires the agent to recognize that the specific transition function and/or the reward function is portable to new situations.…”

“…While the presented reward-predictive model transfers state abstractions across tasks, this model has to re-learn how individual latent states are associated with one-step rewards or SFs for each task. In fact, the presented abstraction transfer models could be combined with prior work [ 3 , 13 , 16 , 31 ] that transfers SFs, latent transition functions, or latent reward functions to integrate the benefits of each transfer system.…”

“…Our work motivates the development of targeted experimental designs that would test if human subjects can reuse a latent structure that is present in a set of tasks despite variations in transitions and rewards. For example, one could design a human subject study similar to [16] where participants solve a sequence of grid-world navigation problems, but augment the design to test if subjects reuse a latent structure present in a set of tasks despite variations in transitions and rewards, similar to the task sequence presented in Fig 5 . As illustrated in Fig 6, the specific pattern of generalization across tasks is predicted to vary depending on whether agents use reward-predictive state abstractions or re-use SR abstractions. Thus, our work provides a concrete testable behavioral prediction that would discriminate between our work and existing work.…”

“…In contrast, this article shows that reward-predictive state abstractions afford “zero-shot” transfer across tasks with variations in transition functions, reward functions, and optimal policies and do not have to be adjusted or re-learned for each task. Such “deep transfer” across environments, even in the absence of prior experience with specific transition or reward functions, is predicted by behavioral and neural signatures of human structure learning [ 4 , 15 , 16 ] but not afforded by alternative algorithms that compress the transition function itself directly [ 17 , 18 ].…”

Reward-predictive representations generalize across tasks in reinforcement learning

“…Our work motivates the development of targeted experimental designs that would test if human subjects can reuse a latent structure that is present in a set of tasks despite variations in transitions and rewards. For example, one could design a human subject study similar to [16] where participants solve a sequence of grid-world navigation problems, but augment the design to test if subjects reuse a latent structure present in a set of tasks despite variations in transitions and rewards, similar to the task sequence presented in vary depending on whether agents use reward-predictive state abstractions or re-use SR abstractions. Thus, our work provides a concrete testable behavioral prediction that would discriminate between our work and existing work.…”