Zooming in on what counts as core and auxiliary: A case study on recognition models of visual working-memory

Abstract

In response to the reported replication crisis in psychology, much recent work focused on increasing the rigor of theory assessment in the social sciences. This research highlights that testing theories is challenging because they inherit a new set of auxiliary assumptions as soon as they are linked to a specific methodology. In this article, we integrate and build on this line of work by demonstrating the breadth of these challenges. In particular, we show that tracking auxiliary assumptions is extremely difficult because they are made at different stages of theory testing and at multiple levels of a theory. We focus on these issues in a reanalysis of a seminal study and its replications, both of which use a simple working-memory paradigm and a mainstream computational modeling approach. These studies provide the main evidence for ‘discrete-slot’ recognition models of visual working-memory, and are still used as the basis for how to measure performance in popular visual working-memory tasks. In our reanalysis, we findthat core practical auxiliary assumptions were unchecked and violated; the original model comparison metrics and data were not diagnostic in several experiments. Furthermore, we find that models were not matched on their ‘theory-general’ auxiliary assumptions, indicating that the tested models were restricted, and not matched in theoretical scope. After testing these auxiliaryassumptions and identifying diagnostic testing conditions, we find evidence for the opposite conclusion. That is, continuous resource models outperform discrete-slot models. Together, our work demonstrates why tracking and testing auxiliary assumptions is a fundamental challenge, even in prominent studies led by careful, computationally-minded researchers. Our work also serves as a framework and conceptual guide for scientists and research consumers on how toidentify and test the gamut of auxiliary assumptions in psychological theory assessment.

You cannot “count” how many items people remember in visual working memory: The importance of signal detection–based measures for understanding change detection performance.

“…We note here that these differences in memory estimates are based on the metrics of each measure obtained from direct transformations of the data, with no model fitting. Accordingly, these metrics are not subject to common criticisms regarding differential model flexibility or overfitting (unlike the results of e.g., Rouder et al, 2008; which appear to arise from the particular assumptions used in the model fits: Robinson et al, 2022).…”

“…In theory, this could cause memory to look more continuous when it is actually discrete (e.g., Rouder et al, 2008). Thus, to test this hypothesis and to thoroughly explore the dichotomy between discrete and continuous memories, in Experiment 3, we replicated Experiment 2 but followed the methods of Rouder et al (2008)-one of the few articles claiming evidence for threshold-like performance (though see Robinson et al, 2022)-in adding a visual mask before the change detection test.…”

“…However, visual working memory researchers have often avoided collecting confidence-based ROC data and instead look to manipulate response bias by changing the prior probability of a "same" vs. "change" response (Donkin et al, 2014(Donkin et al, , 2016Rouder et al, 2008; though see Robinson et al, 2020;Xie & Zhang, 2017). While results from response bias manipulations used to measure ROCs have variedembracing both threshold and signal detection views at different times (e.g., Donkin et al, 2014Donkin et al, , 2016Rouder et al, 2008)-our own recent work suggests this is largely because the data in those studies are not particularly diagnostic (e.g., being very limited in their range of response bias values) and because the model comparison metrics used by the studies were not validated to ensure that they adequately recover the correct model when using simulated data (Robinson et al, 2022). By contrast, data from confidencebased ROCs of change detection in working memory are unequivocal: ROCs have always been found to be curvilinear and most consistent with equal variance signal detection models (Robinson et al, 2020;Wilken & Ma, 2004; see also Xie & Zhang, 2017, who find visually equal variance curves but do not test this class of model directly).…”

“…Note that it is possible to test other aspects of the K model simultaneously with testing its shape, like how fixed it is across set sizes (as done byRouder et al, 2008). However, this confounds both aspects of the model-whether ROCs are linear or curvilinear, and whether performance drops as expected across set sizes(Robinson et al, 2022)-and what we are interested in is the shape of the ROC within a set size, because this is what decides whether the K metric, the d 0 metric, or neither are valid.6 WILLIAMS, ROBINSON, SCHURGIN, WIXTED, AND BRADYThis document is copyrighted by the American Psychological Association or one of its allied publishers.This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.…”

“…Note that it is possible to test other aspects of the K model simultaneously with testing its shape, like how fixed it is across set sizes (as done byRouder et al, 2008). However, this confounds both aspects of the model-whether ROCs are linear or curvilinear, and whether performance drops as expected across set sizes(Robinson et al, 2022)-and what we are interested in is the shape of the ROC within a set size, because this is what decides whether the K metric, the d 0 metric, or neither are valid.UNDERSTANDING CHANGE DETECTION PERFORMANCE…”