Reward-predictive representations generalize across tasks in reinforcement learning

Abstract

One question central to reinforcement learning is which representations can be generalized or re-used across different tasks. Existing algorithms that facilitate transfer typically are limited to cases in which the transition function or the optimal policy is portable to new contexts, but achieving "deep transfer" characteristic of human behavior has been elusive. This article demonstrates that model reductions that minimize error in predictions of reward outcomes generalize across tasks with different transition and reward functions. Such state representations compress the state space of a task into a lower dimensional representation by combining states that are equivalent in terms of both the transition and reward functions. Because only state equivalences are considered, the resulting state representation is not tied to the transition and reward functions themselves and thus generalizes across tasks with different reward and transition functions. The presented results motivate further experiments to investigate if humans or animals learn such a representation, and whether neural systems involved in state representation reflect the modeled equivalence relations.

Gaze-Related Activity in Primate Frontal Cortex Predicts and Mitigates Spatial Uncertainty

“…Based on our model ( Fig. 7 ) and the general principle of reward/effort maximization 57,58 , we propose the following explanation for our neurophysiological data. Our previous results suggest that the FEF and SEF continue to show an eye-centered target-relative-to-eye to gaze-relative-to-eye transformation for saccades in the presence of a landmark 15,16 , but their visual signals are influenced by landmarks in a fashion that depends on target-landmark configuration 32 .…”

“…Note that individual trials are directed pseudorandomly (as in our data) but the overall gaze distribution maximizes reward across trials. Overall, this strategy maximizes reward outcome based on visual cues and their link to expected probabilistic events 57,58 . In lay terms, the model makes an educated ‘guess’.…”

Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for flexible reinforcement learning

“…Previous work has also cast the learning of latent task components and structures as a problem of nonparametric Bayesian inference [22, 10, 11, 12,13,31], and our work continues in this same vein; but Bayesian inference must now be performed over both individual latent components and a potentially infinite variety of structures composed of these components. Bayesian non-parametrics also offers one possible solution to two of the challenges inherent in continual learning.…”

“…On a higher level of abstraction, high-dimensional MDPs, as one finds in sensory-rich naturalistic tasks, can often be compressed to discover regions of state spaces that are equivalent for the sake of planning. Discovery of such lower dimensional latent states allows an agent to transfer to novel MDPs with entirely novel reward and transition functions, as long as they retain the abstract latent structure [13]. Indeed, this work showed that one could discover useful abstractions in a guitar playing task (e.g., that all fret positions that yield a “C” note are equivalent) that allows an agent to much more rapidly learn new scales.…”

“…This framework has offered an account for how humans are able to generalize over multiple structures without interfering with each other. This framework was originally applied to modelling simple context-dependent Pavlovian conditioning [10], but has been extended to capture learning and generalization of entire instrumental "task sets" of stimulus-response mappings [11,1], to predictive representations of future states in sequential decision-making tasks [12], and to deeper latent state abstractions in which new tasks can be learned by analogy with previous tasks [13]. Models of hierarchically nested cortico-striatal gating circuits have offered a mechanistic account for how the brain might approximate such structure learning and transfer [14,11], and human studies have found signatures of latent task structures in the brain [15,16,1,17,2,18].…”