Quantifying the informational value of classification images

Abstract

Reverse correlation is an influential psychophysical paradigm that uses participant’s responses to randomly varying images to build a Classification Image (CI), which is commonly interpreted as a visualization of a participant’s mental representation. It is unclear, however, how to statistically quantify the amount of signal present in CIs, which limits the interpretability of these images. In this paper, we propose a novel metric, infoVal, which assess informational value relative to a resampled random distribution, and can be interpreted as a z-score. In the first part, we define the infoVal metric and show, through simulations, that it adheres to typical Type I error-rates rated under various task conditions (internal validity). In the second part, we show that the metric correlates with markers of data quality in empirical RC data, such as the subjective recognizability, objective discriminability and test-retest reliability of CIs (convergent validity). In the final part, we demonstrate how the infoVal metric can be used to compare the informational value of reverse correlation datasets, comparing data acquired online versus data acquired in a controlled lab environment. We recommend a new standard of good practice where researchers assess infoVal scores of reverse correlation data to ensure that they do not read signal in CIs, where no signal is present. The infoVal metric is implemented in the open source rcicr R-package to facilitate its adoption. This work has now been published in Behavior Research Methods, and can be found here: https://link.springer.com/article/10.3758%2Fs13428-019-01232-2

Dominance and competence face to face: Dissociations obtained with a reverse correlation approach

“…Although attempts have been made to overcome this limitation (see Brinkman et al, 2018) the current use of noise-based RC is still 'hampered by the lack of methodological work addressing validity, reliability, and guidelines for best practice' (Brinkman et al, 2017, p. 352). We must take two aspects into consideration to discuss it.…”

“…The first is related to a limitation of the field itself regarding power implications for the reliability of the CIs (i.e., classification images). Although attempts have been made to overcome this limitation (see Brinkman et al, 2018) the current use of noise-based RC is still 'hampered by the lack of methodological work addressing validity, reliability, and guidelines for best practice' (Brinkman et al, 2017, p. 352). Until an optimal power analytical approach is established, the most viable option is to adopt the task parameters used in previous RC studies, as we have done here.…”

“…Until an optimal power analytical approach is established, the most viable option is to adopt the task parameters used in previous RC studies, as we have done here. A suggestion advanced by Brinkman et al (2018), after our data collection, was that researchers should include at least 500 trials in a RC task to obtain reliable individual CIs (measured with a new index, infoVal, which assesses the amount of signal in a CI relative to CI data derived from random responses). But only future approaches would show how this is relevant to studies like ours that rely on CIs generated from 6,000 trials 2 provided by 20 different individuals.…”

“…Following the recommendations of previous RC studies (Dotsch & Todorov, 2012), twenty participants participated in each condition, and each participant provided 300 decisions (i.e., trials). Although this is not recommended to analyze CIs at an individual level (see Brinkman et al, 2018), our focus was on the socalled group-level CIs-henceforth simply referred to as CIs (for CIs analyses at the individual level, please see sample size requirements presented in Brinkman et al, 2018). We followed a common approach in RC studies and analyzed data at the group-level (Dotsch & Todorov, 2012;Imhoff, Dotsch, Bianchi, Banse, & Wigboldus, 2011).…”

Improving reverse correlation analysis of faces: Diagnostics of order effects, runs, rater agreement, and image pairs

“…A third approach has been to examine whether pixel patterns of the CI generated by the choices over pairs of face images were different from those to be expected with random choices. Brinkman et al (2019) introduced the InfoVal statistic (see also Schmitz, Rougier, & Yzerbyt, 2019;Schmitz, Rougier, Yzerbyt, Brinkman, & Dotsch, 2020) to represent the pixel pattern of the aggregated "noise matrix" that is superimposed on a base face to create a classification image, and suggested it was unlikely that obtained CIs in a variety of reverse correlation studies had been the result of random choices by participants.…”