Not-so-working Memory: Drift in Functional Magnetic Resonance Imaging Pattern Representations during Maintenance Predicts Errors in a Visual Working Memory Task

Abstract

Working memory (WM) is critical to many aspects of cognition, but it frequently fails. Much WM research has focused on capacity limits, but even for single, simple features, the fidelity of individual representations is limited. Why is this? One possibility is that, because of neural noise and interference, neural representations do not remain stable across a WM delay, nor do they simply decay, but instead, they may “drift” over time to a new, less accurate state. We tested this hypothesis in a functional magnetic resonance imaging study of a match/nonmatch WM recognition task for a single item with a single critical feature: orientation. We developed a novel pattern-based index of “representational drift” to characterize ongoing changes in brain activity patterns throughout the WM maintenance period, and we were successfully able to predict performance on the match/nonmatch recognition task using this representational drift index. Specifically, in trials where the target and probe stimuli matched, participants incorrectly reported more nonmatches when their activity patterns drifted away from the target. In trials where the target and probe did not match, participants incorrectly reported more matches when their activity patterns drifted toward the probe. On the basis of these results, we contend that neural noise does not cause WM errors merely by degrading representations and increasing random guessing; instead, one means by which noise introduces errors is by pushing WM representations away from the target and toward other meaningful (yet incorrect) configurations. Thus, we demonstrate that behaviorally meaningful drift within representation space can be indexed by neuroimaging.

Drifting codes within a stable coding scheme for working memory

“…At the neural level, evidence for drift has been found in the neural population code in monkey prefrontal cortex (PFC) during a spatial WM task [15], in which trial-wise shifts in the neural tuning profile predicted whether recall error was clockwise (CW) or counterclockwise (CCW) relative to the correct location. Recently, a human functional MRI (fMRI) study has found that delay activity reflected the probe stimulus more when participants erroneously concluded that it matched the memory item [16], which is consistent with the drift account.…”

Deep-Learning-Based Multivariate Pattern Analysis (dMVPA): A Tutorial and a Toolbox

“…(For example, averaging ERPs with different latencies can produce a grand average that appears lower in both amplitude and frequency than any individual trial's ERP actually is). In our own prior work, we have most certainly found that MVPA enabled us to address questions that could not be answered with previous univariate approaches; for example, that scene-selective visual brain areas represented not only category but exemplarlevel information during mental imagery (Johnson and Johnson, 2014), that ERPs related to mental attention that did not exhibit reliable differences between stimulus categories at a grandaverage level still contained category information with MVPA (Johnson et al, 2015), and that gradual drift in fine-grained information patterns in visual cortex during working memory could be used to predict memory errors (Lim et al, 2019).…”

“…Numerous MVPA variations exist, including those based on correlation (either Pearson or rank-based; Haxby et al, 2001), support vector machines (SVMs; De Martino et al, 2008;Dosenbach et al, 2010), logistic regression (Akama et al, 2012), sparse multinomial logistic regression (SMLR; Krishnapuram et al, 2005;Kohler et al, 2013), naïve Bayes classifiers (Kassam et al, 2013), and more. Many of these techniques concern classification of brain patterns into discrete cognitive states, whereas others examine different aspects of the data (e.g., overall similarity between brain patterns; Xue et al, 2010;Lim et al, 2019) without explicit categorization, but all of them represent increases in mathematical and conceptual sophistication over univariate techniques. Importantly, when compared to earlier univariate techniques, MVPA has enabled us to examine in a much more nuanced fashion how brain activity patterns encode mental states.…”

Deep-Learning-Based Multivariate Pattern Analysis (dMVPA): A Tutorial and a Toolbox

“…Numerous MVPA variations exist, including those based on correlation (either Pearson or rank-based; Haxby et al, 2001), support vector machines (SVMs; De Martino et al, 2008;Dosenbach et al, 2010), logistic regression (Akama et al, 2012), sparse multinomial logistic regression (SMLR; Kohler et al, 2013;Krishnapuram et al, 2005), naïve Bayes classifiers (Kassam et al, 2013), and more. Many of these techniques concern classification of brain patterns into discrete cognitive states, whereas others examine different aspects of the data (e.g., overall similarity between brain patterns; Xue et al, 2010;Lim et al, 2019) without explicit categorization, but all of them represent increases in mathematical and conceptual sophistication over univariate techniques. Importantly, when compared to earlier univariate techniques, MVPA has enabled us to examine in a much more nuanced fashion how brain activity patterns encode mental states.…”