Improving calibration without training: the role of task information

Abstract

Medical students estimated probabilities that medical school applicants selected randomly from a defined population would be offered a place on the basis of information about eight characteristics of each one. Estimates were biased in favour of acceptance and this was unaffected by mere provision of base rate information. However, a first experiment showed that provision of this information reduced miscalibration after each candidate had been discussed within small groups of participants. A second experiment showed beneficial effects of providing people with the range of probability responses derived from a statistical model of how candidates' characteristics influenced whether or not they were offered a place. Numerical and graphical representations of this information both improved calibration but did so by having different effects on the distribution of participants' probability responses.

Demonstrating and eliminating the aggregation effect in physical skill tasks

“…Instead, as recently argued by Rakow, Harvey, and Finer (2003), base rates must be presented in a manner that will effectively induce participants to consider them. Therefore, instead of providing one overall base rate to our participants, we provided the base rate (of successful tosses) for each of the 10 different toss locations that comprised our task.…”

The recalcitrance of overconfidence and its contribution to decision aid neglect

“…Nevertheless, we thought that by highlighting those trials in which the equation and decision maker disagreed on the first block of trials, we would thereby be concentrating on those instances in which improvement in calibration would have a very large potential benefit. Rakow, Harvey, and Finer (2003) provided medical students with important base-rate information to help them decide which of a sample of 36 medical school applicants were and were not admitted. The authors found that some medical students did not think that the base rate of acceptance of this sample was representative of the population, and other medical students simply did not take sufficient account of whatever base rate they thought was accurate.…”

Testing Best Practices to Reduce the Overconfidence Bias in Multi-criteria Decision Analysis

“…Several studies indeed demonstrated the positive effect of feedback on reduction in overconfidence in some cases [48,49]. However this effect is susceptible to various factors, such as task difficulty [50], estimation order (i.e., first estimation vs. following estimations, see Baranski and Petrusic [ 51 ], Lichtenstein and Fischhoff [52]), and type of judgment (i.e., high probability vs. low probability, see Baranski and Petrusic [51]).…”