Current and future goals are represented in opposite patterns in object-selective cortex

Abstract

Adaptive behavior requires the separation of current from future goals in working memory. We used fMRI of object-selective cortex to determine the representational (dis)similarities of memory representations serving current and prospective perceptual tasks. Participants remembered an object drawn from three possible categories as the target for one of two consecutive visual search tasks. A cue indicated whether the target object should be looked for first (currently relevant), second (prospectively relevant), or if it could be forgotten (irrelevant). Prior to the first search, representations of current, prospective and irrelevant objects were similar, with strongest decoding for current representations compared to prospective (Experiment 1) and irrelevant (Experiment 2). Remarkably, during the first search, prospective representations could also be decoded, but revealed anti-correlated voxel patterns compared to currently relevant representations of the same category. We propose that the brain separates current from prospective memories within the same neuronal ensembles through opposite representational patterns.

Different states of priority recruit different neural representations in visual working memory

“…For this reason, we characterize the priority-based transformations reported here as examples of remapping, not of recoding. This remapping account is also consistent with the findings of van Loon and colleagues [19], who observed that a classifier trained on a stimulus category when it was the PMI could also decode that category when it was the UMI, even though the pattern of activity of the UMI projected into an opposite region of multidimensional scaling space relative to the PMI. The characteristics underlying priority-based remapping may account for some of the mixed results in previous work on the effects of retrocuing.…”

“…When a retrocue designated an item a UMI, IEM indicated that its representation in early visual cortex underwent a transformation that corresponded to a rotation of 90˚relative to how it was represented in the FoA. By extension, the present results also lend support to the interpretation of the priority-based transformation reported by van Loon and colleagues [19]. Because they observed a transformation of the posterior inferior temporal representation of objects in their study (cows, dressers, ice skates) to an "opposite" pattern in multidimensional scaling space, the general principle underlying priority-based remapping may be one of transforming the representation of deprioritized information into a format that is complementary to its representation when in the FoA, effectively maximizing the difference between an item when it is a UMI versus when it is a PMI.…”

“…First, it is unclear whether this operation can generalize to different domains of information, for example, low-level visual information. Second, the van Loon and colleagues study [19] observed the opposite pattern only during the search display. Tracking the changes in the neural representations throughout the delay period (with no stimulus on screen) in a typical working memory task would be essential to understand the mechanism of this neural signal.…”

“…Recently, a third alternative, positing a representational transformation of the UMI, has been proposed. One source of this idea was the results from a dual serial visual search task, in which the pattern of activity representing a search template in object-selective posterior fusiform cortex transitioned to an "opposite" format if a different template was to be searched for prior to the critical template (i.e., when the critical template was the UMI) relative to when it was relevant for the current search [19]. By this representational transformation scheme, both the UMI and the PMI may be encoded simultaneously in posterior cortex, with the format of their neural representation varying with priority status [20].…”

“…Although the van Loon and colleagues [19] study provided a new and plausible account for the representation of the UMI, much remains to be explored for this priority-based transformation account. First, it is unclear whether this operation can generalize to different domains of information, for example, low-level visual information.…”