Calibration and probability judgements: Conceptual and methodological issues

Keren, Gideon

doi:10.1016/0001-6918(91)90036-y

Cited by 494 publications

(333 citation statements)

References 74 publications

Supporting

Mentioning

319

Contrasting

Unclassified

Order By: Relevance

“…Our forecasters were not equally overconfident across the range of confidence. Figure 1 divides confidence into bins, as is common practice (Keren 1991). The most striking result is how well calibrated forecasters were: the dots lie close to the identity line.…”

Section: Resultsmentioning

confidence: 88%

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Swift

Minster²,

Mellers³

et al. 2016

Preprint

View full text Add to dashboard Cite

This research examines the development of confidence and accuracy over time in the context of forecasting. Although overconfidence has been studied in many contexts, little research examines its progression over long periods of time or in consequential policy domains. This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events. We sought to apply insights from research to structure the questions, interactions, and elicitations to improve forecasts. Indeed, forecasters’ confidence roughly matched their accuracy. As information came in, accuracy increased. Confidence increased at approximately the same rate as accuracy, and good calibration persisted. Nevertheless, there was evidence of a small amount of overconfidence (3%), especially on the most confident forecasts. Training helped reduce overconfidence and team collaboration improved forecast accuracy. Together, teams and training reduced overconfidence to 1%. Our results provide reason for tempered optimism regarding confidence calibration and its development over time in consequential field contexts.

show abstract

Section: Resultsmentioning

confidence: 88%

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Swift

Minster²,

Mellers³

et al. 2016

Preprint

View full text Add to dashboard Cite

show abstract

“…Nevertheless, these studies often report that confidence exceeds accuracy. For example, when subjects express 90% confidence, they may be correct only about 75% of the time (for reviews, see Keren, 1991;Lichtenstein, Fischhoff, &Phillips, !982;Yates, 1990). Overconfidence is prevalent but not universal: It is generally eliminated and even reversed for very easy items.…”

Section: Overconfidencementioning

confidence: 99%

On the reality of cognitive illusions.

Kahneman

Tversky²

1996

Psychological Review

1,126

452

View full text Add to dashboard Cite

The study of heuristics and biases in judgment has been criticized in several publications by G. Gigerenzer, who argues that "biases are not biases" and "heuristics are meant to explain what does not exist" (1991, p. 102). This article responds to Gigerenzer's critique and shows that it misrepresents the authors' theoretical position and ignores critical evidence. Contrary to Gigerenzer's central empirical claim, judgments of frequency-not only subjective probabilities-are susceptible to large and systematic biases. A postscript responds to Gigerenzer's (1996) reply.

show abstract

“…External correspondence is the degree to which a person's likelihood judgments accord with the actual occurrence or nonoccurrence of the target events under consideration (Yates, 1982). The literature on overconfidence, for example, is concerned with one aspect of external correspondence (e.g., Adams & Adams, 1961;Fischhoff, Slovic, & Lichtenstein, 1977;Lichtenstein, Fischhoff, & Phillips, 1982;Keren, 1991). Do people's judgments of the likelihood of some target event "match" the relative frequency with which the target event occurs?…”

Section: External Correspondencementioning

confidence: 99%

Effects of a Relative-Frequency Elicitation Question on Likelihood Judgment Accuracy: The Case of External Correspondence

Price

1998

Organizational Behavior and Human Decision Processes

View full text Add to dashboard Cite

In three experiments, college students judged the likelihood that they chose the correct alternative for each of 40 two-alterna-tive, generalknowledge items. They responded either to a rela-tive-frequency elicitation question ("Out of 100 questions for which you felt this certain of the answer, how many would you answer correctly?") or to a probability elicitation question ("What is the probability that you chose the correct answer?"). Judg-ments in response to the relative-frequency elicitation question tended to be lower, exhibit less scatter, and express complete certainty less often than judgments in response to the probability elicitation question. Two types of explanation for these effects are considered. First, the effect of the relative-frequency elicitation question may be to reduce random response error in participants' likelihood judgments. Second, the relativefrequency elicitation question may encourage the use of frequency information and simpler algorithms for making likelihood judgments.Behavioral decision researchers have shown repeatedly that the way one person asks another for a judgment can have important effects on the judgment itself. These effects are typically referred to as framing effects (Yates, 1990). The present research concerns one such framing effect in the domain of likelihood judgment. Specifically, it concerns the effects of a relative-frequency elicitation question (e.g., "In 100 cases like this one, how many times would target event

show abstract

Calibration and probability judgements: Conceptual and methodological issues

Cited by 494 publications

References 74 publications

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

On the reality of cognitive illusions.

Effects of a Relative-Frequency Elicitation Question on Likelihood Judgment Accuracy: The Case of External Correspondence

Contact Info

Product

Resources

About