Decision noise may mask criterion shifts: Reply to Balakrishnan and MacDonald (2008)

Abstract

(2008) argue that RTbased measures of signal detection processes provide evidence against signal detection theory's notion of a flexible decision criterion. They argue that this evidence is immune to the alternative explanation proposed by S. T. Mueller and C. T. Weidemann (2008), that decision noise may mask criterion shifts. We show that noise in response times can produce the same effects as are produced by noise in confidence ratings. Given these results, the evidence is not sufficient to categorically reject the notion of a flexible response policy implemented through shifts in a decision criterion.Theories of signal detection attempt to distinguish perceptual factors from decision effects (such as response biases). To that effect, signal detection theory (SDT) incorporates the notion of a flexible decision criterion that adapts to contingencies in the environment (such as stimulus base rates or pay-off schemes). Balakrishnan (1998aBalakrishnan ( , 1998bBalakrishnan ( , 1999) identified serious problems with SDT that render derived measures of sensitivity and bias (such as d and ) suspect. On the basis of novel measures of bias, Balakrishnan (1998aBalakrishnan ( , 1998bBalakrishnan ( , 1999 argued that the notion of such a flexible decision criterion that shifts in response to stimulus contingencies is fundamentally flawed.We have shown that Balakrishnan's (1998a) results, as well as those from other experiments, can in fact be accounted for by a simple model incorporating a flexible decision criterion (Mueller & Weidemann, 2008). This model is an extension of SDT, but in addition to perceptual noise, it includes noise in the mapping from percepts to responses (decision noise). In particular, we distinguish between two types of decision noise: classification noise and confidence noise. Classification noise refers to noise in the mapping between the percept and the classification response (e.g., yes-no, present-absent, A-B), whereas confidence noise refers to the noise in the mapping between an internal state and a dependent measure indexing response confidence (such as confidence ratings). We view the decision noise model (DNM) as constituting a simple existence proof (rather than a full-fledged alternative to existing theories) that the results from signal detection tasks 1 are compatible with flexible decision criteria that adapt to task contingencies. In the DNM, confidence noise can be larger than classification noise, which is consistent with the results of our receiver operating characteristic (ROC) analyses comparing response-related distalstimulus ROC functions and confidence ROC functions (see Mueller & Weidemann, 2008, for details). As we demonstrated earlier (Mueller & Weidemann, 2008), this discrepancy between classification noise and confidence noise can mask shifts in decision criteria.Balakrishnan and MacDonald (2008) criticize our work on the basis of three major points. Specifically, they argue that:1. Our account of how decision noise may mask criterion shifts should not apply to cases w...

Decision criteria do not shift: Commentary on Mueller and Weidemann (2008)

“…Mueller and Weidemann (2008) propose that these unexpected phenomena are not due to problems with the decision-criterion construct but are instead due to two compounded effects: instability of the decision criterion across trials, and even greater instability in the flanking criteria that determine which confidence rating will be reported. There Mueller and Weidemann's experiment. In an Addendum, we respond to Weidemann and Mueller's (2008) reply to this Comment. In many areas of perception and memory research, experimental phenomena that appear to have significant implications about perception or memory processes per se could actually be due to the effects of response biases. For the past 50 years or so, the most widely accepted method of distinguishing these two possibilities has been to apply a signal detection analysis to the data (e.g., Green & Swets, 1966).…”

“…In their reply to this commentary, Weidemann and Mueller (2008) have suggested, first, that confidence ratings and RT are both "noisy indices" of perceptual evidence. This hypothesis is self-evident, but it does not resolve any of the issues.…”

Assessing recognition memory using confidence ratings and response times

“…In the limit, the partitions of this variable within a given classification response contain equal proportions of correct and incorrect responses such that the rate of increase of hits and FAs in the ROC function is constant within each classification response. Thus, a ‘random ROC’ that reveals no information beyond that contained in the classification response is bilinear, connecting the origin to the classification point and that point to (1,1) with straight lines (shown together with the main diagonal as dotted lines in figure 1 ; [ 8 , 24 , 45 ]). The AUC therefore conflates classification performance with the measure’s ability to reflect the signal underlying the classification decision.…”

Decision criteria in dual discrimination tasks estimated using external-noise methods

“…The estimation of these internal parameters, as well as of d′ and c, relies on some critical assumptions regarding the statistics of the internal events and the dynamics of the associated decision mechanism. Thus, not surprisingly, the validity of the assumptions made and their associated consequences for the obtained measure of sensitivity are under extensive research (Balakrishnan, 1998a(Balakrishnan, , 1998b(Balakrishnan, , 1999Balakrishnan & MacDonald, 2002Treisman, 2002;Weidemann & Mueller, 2008).…”