Comparing memory capacity across stimuli requires maximally dissimilar foils: Using deep convolutional neural networks to understand visual working memory capacity for real-world objects

Abstract

The capacity of visual working memory has been subject to considerable debate: whether there is a relatively fixed item limit, regardless of what these items are; or a fixed resource limit; or whether capacity limits vary depending on the complexity or familiarity of items. Here, we argue that before asking the question of how capacity varies across different stimuli, it is necessary to establish a methodology that allows a fair comparison between distinct stimulus sets. One of the most important factors determining performance in a memory task is target/foil dissimilarity. Thus, we argue that only by maximizing the dissimilarity of the target and foil in each stimulus set can we provide a fair basis for memory comparisons. In the current work we introduce a new way to pick such foils objectively for complex, meaningful real-world objects by using deep convolutional neural networks that are sensitive to both semantic and visual features. Using this method, we resolve an important theoretical issue in the visual working memory literature by showing that there is a greater capacity for real-world objects relative to simple colors; critically, we also show that this difference can be reduced or eliminated when non-comparable foils are used, potentially explaining why previous work has not always found such a difference. Our study thus demonstrates that working memory capacity is not fixed capacity but depends critically on the type of information that is remembered, and offers a solution of how to compare memory performance and other cognitive systems across different stimulus sets on common ground.

Target templates in low target-distractor discriminability visual search have higher resolution, but the advantage they provide is short-lived

“…Deep convolutional neural networks are useful models of human recognition and the human visual system (e.g., Yamins et al, 2014 ), and deep nets trained on categorization are sensitive to some extent to both visual and semantic features (e.g., Jozwik et al, 2017 ; Peterson et al, 2018 ). Such measures also reliably predict memory confusability (Brady & Störmer, 2020 ). Here, we found they captured the categorical structure of the individual images, and that they indicated that artifacts would be more similar to the search targets than the other stimuli, providing further validation of our stimulus design.…”

“…Thus, to estimate the pairwise image similarity, we made use of a pre-trained deep convolutional neural network (CNN). CNNs are useful metrics of object similarity for predicting memory (Brady & Störmer, 2020 ), and have consistently been shown to provide some level of match to the human visual system (Yamins et al, 2014 ). As such models are trained to perform categorization, the high-level features in them contain both semantic and visual information, but are primarily visual in the sense that the network has no broader conceptual knowledge beyond categorical classification.…”

Real-world objects are not stored in holistic representations in visual working memory

“…For example, Brady et al (2016) showed a boost in performance for real-world objects that was attributable to more active storage in visual working memory, consistent with a theory where additional high-level information about such objects, perhaps in the ventral stream, is maintained in working memory in addition to low-level information. Some recent studies (Li, Xiong, Theeuwes, & Wang, 2020;Quirk, Adam, & Vogel, 2020) instead found no difference between storing simple features and real-world objects in visual working memory, but these results were likely due to a lack of control for similarity between targets and foils in the color versus real-world object tasks (Brady & Störmer, 2020;Brady & Störmer, in press). With better control for target-foil similarity (Brady & Störmer, 2020), real-world objects result in significantly better performance compared with simple features (Brady & Störmer, 2020;Brady & Störmer, in press).…”

“…Looking at the performance on the exemplar task compared with the exemplar-state task, one could argue that there are differences in the exemplar and state manipulations themselves that account for this effect. For example, it intuitively seems that two different states of the same object might be more visually similar than two different exemplars in the same state, and that this could affect the two-alternative forced choice task performance (e.g., Brady & Störmer, 2020). In this case, the exemplar-state task would be harder than the exemplar task based on the images alone, rather than because of binding difficulties.…”

“…The examples of most similar and dissimilar pairs of exemplars and states according to layer 1 (F). similarity using the VGG16 pretrained convolutional neural network (Brady & Störmer, 2020;Simonyan & Zisserman, 2014). We were particularly focused on the top max-pool layer, which allowed us to retrieve high-level features that are more invariant to low-level transforms, because previous work demonstrates that this provides a useful proxy for object similarity (Brady & Störmer, 2020) with extremely similar object stimuli.…”