Measuring memory is harder than you think: How to avoid problematic measurement practices in memory research

Abstract

We argue that critical areas of memory research rely on problematic measurement practices and provide concrete suggestions to improve the situation. In particular, we highlight the prevalence of memory studies that use tasks (like the "old/new" task: "have you seen this item before? yes/no") where quantifying performance is deeply dependent on counterfactual reasoning that depends on the (unknowable) distribution of underlying memory signals. As a result of this difficulty, different literatures in memory research (e.g., visual working memory, eyewitness identification, picture memory, etc) have settled on a variety of fundamentally different metrics to get performance measures from such tasks (e.g., A′, corrected hit rate, percent correct, d′, diagnosticity ratios, K values, etc.), even though these metrics make different, contradictory assumptions about the distribution of latent memory signals, and even though all of their assumptions are frequently incorrect. We suggest that in order for the psychology and neuroscience of memory to become a more cumulative, theory-driven science, more attention must be given to measurement issues. We make a concrete suggestion: the default memory task for those simply interested in performance should change from old/new ("did you see this item'?") to two-alternative forced-choice ("which of these two items did you see?"). In situations where old/new variants are preferred (e.g., eyewitness identification; theoretical investigations of the nature of memory signals), receiver operating characteristic (ROC) analysis should be performed rather than a binary old/new task.

Decoding the content of working memory in school-aged children

“…The contrast between paradigms with and without secondary tasks has the potential to upend common conceptions of WM development and refreshing as a maintenance mechanism. Combined with the results regarding the organization strategy, on a larger scale, these outcomes (as well as e.g., Brady et al, 2021) highlight the need for deeper considerations of methodology before testing theoretical assumptions in the WM field.…”

Gamified inoculation interventions do not improve discrimination between true and fake news: Reanalyzing existing research with receiver operating characteristic analysis.

“…ROC analysis like that we have recommended here has the potential to identify more nuanced response bias problems. Indeed, some theorists have gone so far as to suggest that unless ROC curves are used, it is impossible to measure memory efficacy in such common paradigms as old/new recognition (see Brady et al, 2022). In the context of eyewitness memory, Mickes et al (2012) introduced ROC analysis to examine the relative efficacy of sequential versus simultaneous lineup procedures.…”

Establishing a causal role for left ventrolateral prefrontal cortex in value-directed memory encoding with high-definition transcranial direct current stimulation

“…The studied words from Session 1 and Session 2 were intermixed during the Day 2 recognition test, but we were interested in whether memory would differ for words encoded during these two sessions as a function of word value. As the standard deviation of the raw ratings of studied words was significantly larger than that of lure words (the average ratio across 63 participants was 1.21 (SD = 0.207) and was significantly greater than 1, t(62) = 8.03, p < 0.001), and to take full advantage of our 6-point recognition confidence scale (which provides more information about memory strength than old/new judgments alone, we conducted a receiver operating characteristic (ROC) curve analysis using the unequal-variance signal-detection (UVSD) model (Brady et al, 2022;Mickes et al, 2007). This UVSD model yields memory sensitivity values (sometimes denoted as da) that are analogous to standard d' values that could be derived from the hit rates and false alarm rates alone, but are considered a superior measure because their derivation takes into account the entire distribution of confidence judgements to old and new items (Brady et al, 2022).…”