Testing models of decision making using confidence ratings in classification.

Abstract

Classification implies decision making (or response selection) of some kind. Studying the decision process using a traditional signal detection theory analysis is difficult for two reasons: (a) The model makes a strong assumption about the encoding process (normal noise), and (b) the two most popular decision models, optimal and distance-from-criterion models, can mimic each other's predictions about performance level. In this article, the authors show that by analyzing certain distributional properties of confidence ratings, a researcher can determine whether the decision process is optimal, without knowing the form of the encoding distributions. Empirical results are reported for three types of experiments: recognition memory, perceptual discrimination, and perceptual categorization. In each case, the data strongly favored the distance-from-criterion model over the optimal model.

Is signal detection theory fundamentally flawed? A response to Balakrishnan (1998a, 1998b, 1999)

“…Calculations were also performed using small samples (10 -50) resampled from experimental data sets. In these data, s L and s H were very similar (e.g., lines of length 6.75 and 6.9 cm, with n 5 6.6 cm), and the two conditions were presented in intermingled short blocks in 2-h sessions with well-practiced subjects (Balakrishnan & Ratcliff, 1996). The previous finding was supported (Balakrishnan, 1998b, Table 3).…”

Multiple confidence estimates as indices of eyewitness memory.

“…Consequently, the fundamentally similar nature of absolute magnitude estimations and the confidence estimates generated by the new procedure may point to a similar insensitivity to context effects for the confidence procedure. Finally, Balakrishnan and Ratcliff (1996) reported little evidence of context effects for a task requiring participants to use confidence ratings to discriminate between and categorize lines of varying lengths. Thus, the extant literature makes no clear predictions about the likelihood of observing context effects for the new procedure.…”

Decision rules for recognition memory confidence judgments.

“…We refer to this model, which is illustrated in the top panel of Figure 3, as the lockstep model because all of the criteria move together. This model has also been referred to as the distance from criterion model because the position of a particular confidence criterion is always a fixed distance from the old-new decision criterion (Balakrishnan & Ratcliff, 1996). The lockstep model is both computationally simple and intuitively plausible.…”

“…Balakrishnan and Ratcliff (1996) recently found that for a strength manipulation the lockstep model (which they termed the "distance from criterion" model) provided a better account of their data than the likelihood ratio model. Their method of analysis differed considerably from the one used here.…”

“…The data reported by Balakrishnan and Ratcliff were visually more in line with the predictions of the lockstep model. Balakrishnan and Ratcliff (1996) compared the predictions of the lockstep model versus an idealized likelihood ratio model, whereas we tested for statistically significant deviations (however small) from the lockstep pattern. Our findings suggest that although the deviations from the lockstep model may not be as extreme as they should be, they are significant and are in the direction predicted by the likelihood ratio account.…”