The Successor Representation and Temporal Context

Abstract

The successor representation was introduced into reinforcement learning by Dayan ( 1993 ) as a means of facilitating generalization between states with similar successors. Although reinforcement learning in general has been used extensively as a model of psychological and neural processes, the psychological validity of the successor representation has yet to be explored. An interesting possibility is that the successor representation can be used not only for reinforcement learning but for episodic learning as well. Our main contribution is to show that a variant of the temporal context model (TCM; Howard & Kahana, 2002 ), an influential model of episodic memory, can be understood as directly estimating the successor representation using the temporal difference learning algorithm (Sutton & Barto, 1998 ). This insight leads to a generalization of TCM and new experimental predictions. In addition to casting a new normative light on TCM, this equivalence suggests a previously unexplored point of contact between different learning systems.

Predicting the past, remembering the future

“…One possibility, borrowing ideas from reinforcement learning, is to directly update an estimate the SR (M̂) from state transitions. Specifically, the SR can be updated incrementally using a form of temporal difference learning [26]:…”

“…According to TCM, items are bound in memory to a slowly drifting representation of temporal context (a recency-weighted average of previous items), and at test the temporal context acts as a retrieval cue, preferentially drawing items based on their strength of association. It can be shown (see [26] for details) that the temporal context representation corresponds to the eligibility trace e t , and the matrix of item-context associations corresponds to M̂.…”

Do we really become smarter when our fluid-intelligence test scores improve?

“…The SR is essentially identical to the fundamental matrix in the theory of Markov chains (Kemeny & Snell, 1976). More recently, Gershman, Moore, Todd, Norman, and Sederberg (2012) identified a formal connection between the SR and an influential model of episodic and semantic memory, the Temporal Context Model (e.g. Howard & Kahana, 2002; Sederberg, Howard, & Kahana, 2008).…”

Compound cuing in free recall.

“…This idea is embodied in retrieved context models of episodic memory such as the temporal context model of Howard and Kahana (2002a) and its more recent variants (e.g., Sederberg et al, 2008; Polyn et al, 2009; Sederberg, Gershman, Polyn, & Norman, 2011; Howard, Kahana, & Wingfield, 2006; Gershman, Moore, Todd, Norman, & Sederberg, 2012). According to these models, the context cue used for recall of items contains a recency-weighted sum of previously activated cognitive states, and as such predict that the temporal contiguity effect should be enhanced when a sequence of previously recalled items were studied at neighboring list positions.…”