Quantifying the informational value of classification images

Abstract

Reverse correlation is an influential psychophysical paradigm that uses a participant’s responses to randomly varying images to build a classification image (CI), which is commonly interpreted as a visualization of the participant’s mental representation. It is unclear, however, how to statistically quantify the amount of signal present in CIs, which limits the interpretability of these images. In this article, we propose a novel metric, infoVal, which assesses informational value relative to a resampled random distribution and can be interpreted like a z score. In the first part, we define the infoVal metric and show, through simulations, that it adheres to typical Type I error rates under various task conditions (internal validity). In the second part, we show that the metric correlates with markers of data quality in empirical reverse-correlation data, such as the subjective recognizability, objective discriminability, and test–retest reliability of the CIs (convergent validity). In the final part, we demonstrate how the infoVal metric can be used to compare the informational value of reverse-correlation datasets, by comparing data acquired online with data acquired in a controlled lab environment. We recommend a new standard of good practice in which researchers assess the infoVal scores of reverse-correlation data in order to ensure that they do not read signal in CIs where no signal is present. The infoVal metric is implemented in the open-source rcicr R package, to facilitate its adoption.Electronic supplementary materialThe online version of this article (10.3758/s13428-019-01232-2) contains supplementary material, which is available to authorized users.

Type I Error Is Inflated in the Two-Phase Reverse Correlation Procedure

“…More objective comparisons of group CIs could reveal whether they exhibit sufficient differences that exceed what we should expect to occur by chance. Recently, Brinkman and colleagues (2020) proposed a promising technique for evaluating the informational value of a single composite. The logic of this test is very similar to a single-sample t test in which an observation is compared against a reference distribution to assess its likelihood of occurrence solely through the operation of a random process.…”

Introducing the Brief Reverse Correlation

“…Additionally, only the individual-level allows more fine-grained analyses (e.g., correlating judgments of the visual rendering with another individual-level variable). However, whereas individual-level visual outcomes are preferable, a large number of trials is usually required to achieve high-quality outcomes, which entails other issues (i.e., economically costly, time demanding, decreased participants' motivation to complete the task in a conscientious manner; Brinkman, Goffin et al, 2019;Todorov et al, 2011). Researchers have therefore noted the need to improve the method in order to "generate higher quality outcomes or reduce the number of trials" (Todorov et al, 2011, p. 787).…”

“…This is because they are composed of much fewer trials than the average CIs (Brinkman et al, 2017;Cone et al, 2020;Jack & Schyns, 2017;Ratner et al, 2014;Todorov et al, 2011). Indeed, RC procedures usually comprise 300-1,000 trials per individuallevel CI (and thus 300-1,000 trials multiplied by the number of participants within a condition BRIEF REVERSE CORRELATION 8 for average-level CIs; Brinkman et al, 2017;Brinkman, Goffin et al, 2019;Cone et al, 2020;. Although adding more trials should in principle enhance the quality of individual CIs, in practice, however, it does not always bear fruit.…”

“…Although adding more trials should in principle enhance the quality of individual CIs, in practice, however, it does not always bear fruit. In fact, doing so may even be detrimental because this hinders participants' motivation to mindfully complete the task (Brinkman, Goffin et al, 2019;Todorov et al, 2011; see also Lick et al, 2013). Furthermore, increasing the length of the task can be time-consuming, economically costly, or simply impractical.…”

“…This would increase the chances of false positives when judging low quality individual CIs as well as the resulting average CIs. Indeed, the RC method will always yield a CI whatever the number of trials or the meaningfulness of responses (Brinkman, Goffin et al, 2019). To address this issue, Brinkman, Goffin et al (2019) developed infoVal, an information value metric that assesses the degree to which an individual CI derives from signal rather than noise.…”