The illusion of knowledge: When more information reduces accuracy and increases confidence

Hall, Crystal C.; Ariss, Lynn; Todorov, Alexander

doi:10.1016/j.obhdp.2007.01.003

Cited by 168 publications

(138 citation statements)

References 32 publications

Supporting

Mentioning

119

Contrasting

Unclassified

Order By: Relevance

“…It is remarkable because the two alternatives are so salient. On the one hand, there are circumstances in which new information increases confidence more than accuracy, exacerbating overconfidence (Hall et al 2007, Heath and Gonzalez 1995, Oskamp 1965. On the other hand, there are other circumstances in which increasing expertise can increase accuracy faster than confidence, reducing overconfidence (Koriat et al 2002).…”

Section: Discussionmentioning

confidence: 99%

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Swift

Minster²,

Mellers³

et al. 2016

Preprint

View full text Add to dashboard Cite

This research examines the development of confidence and accuracy over time in the context of forecasting. Although overconfidence has been studied in many contexts, little research examines its progression over long periods of time or in consequential policy domains. This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events. We sought to apply insights from research to structure the questions, interactions, and elicitations to improve forecasts. Indeed, forecasters’ confidence roughly matched their accuracy. As information came in, accuracy increased. Confidence increased at approximately the same rate as accuracy, and good calibration persisted. Nevertheless, there was evidence of a small amount of overconfidence (3%), especially on the most confident forecasts. Training helped reduce overconfidence and team collaboration improved forecast accuracy. Together, teams and training reduced overconfidence to 1%. Our results provide reason for tempered optimism regarding confidence calibration and its development over time in consequential field contexts.

show abstract

Section: Discussionmentioning

confidence: 99%

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Swift

Minster²,

Mellers³

et al. 2016

Preprint

View full text Add to dashboard Cite

show abstract

“…Tsai et al (in press), who used a sequential information acquisition task, found that judges' confidence in their decisions rose steadily as additional information was presented, even though accuracy did not improve. Under some conditions, in fact, additional knowledge was found to increase confidence while decreasing accuracy (Hall, Ariss, & Todorov, 2007).…”

Section: University Of Haifa Haifa Israelmentioning

confidence: 95%

“…As is often noted, understanding the processes underlying confidence in a belief is important, because confidence affects whether people translate their beliefs into behavior (Gill et al, 1998;Goldsmith & Koriat, 2008;Hall et al, 2007). Indeed, in a recent study, I presented the 105 questions of to 60 participants under freerecognition instructions (see Koriat & Goldsmith, 1996): They would win 1 point for each correct answer and lose 1 point for each incorrect answer, but they could choose not to answer a question, in which case they would neither lose nor win anything (a monetary bonus was promised, depending on the number of points won).…”

Section: Methodsmentioning

confidence: 99%

When confidence in a choice is independent of which choice is made

Koriat

2008

Psychonomic Bulletin & Review

View full text Add to dashboard Cite

), participants chose an answer to two-alternative general-information questions and indicated their confidence in that answer. Whereas previous studies focused on the determinants of the choice and, consequently, on the confidence in that choice, the present study examined a component of subjective confidence that correlates with properties of the items itself, no matter which answer is chosen.The item-based approach to the study of metacognitive judgments has proved useful in the past. According to this approach, some insight into the processes underlying metacognitive judgments and their accuracy can be gained by investigating systematic differences between memory items. This approach is based on the assumption that memory questions differ reliably in characteristics that are pertinent to metacognitive judgments, and that normative data, aggregated across participants, provide clues to the processes that occur within participants. Thus, in Koriat and Lieblich's (1977) study of the tip-of-the-tongue (TOT) states, word definitions were found to differ reliably in terms of two orthogonal dimensions: their likelihood of suggesting the correct memory target, and the extent to which they evoke a feeling of knowing (FOK). The analysis of items in terms of these dimensions provided information about the processes underlying FOK and TOT states, and laid the groundwork for Koriat's (1993) accessibility model of FOK judgments. Similarly, general-information questions were found to differ reliably in two properties among participants who provided an answer, and these properties predicted the magnitude and accuracy of FOK judgments among participants who failed to provide an answer to these questions (Koriat, 1995). Also, using two-alternative general-information questions, has documented a systematic correlation between subjective confidence in the answer and the percentage of participants who chose that answer: Consensually chosen answers elicited stronger confidence than nonconsensual answers, regardless of whether they were correct or wrong, implying that confidence is diagnostic of the correctness of the answer only when the consensually endorsed answer is the correct answer, as is generally the case. When the consensual answer was the wrong answer ("deceptive"; see Fischhoff, Slovic, & Lichtenstein, 1977), this answer was endorsed with stronger confidence. Similar findings were reported by Brewer and Sampaio (2006), who also used an item-based approach to study confidence in memory for sentences.The present study focuses on another item-based correlation that I have observed over the years across several sets of data. Consider the recent study of Koriat (2008) just mentioned: Participants answered 105 questions and expressed their confidence on a 50%-100% scale. The questions had been selected to represent a wide dispersion in terms of the likelihood that participants would choose the correct answer, so that for some questions the consensually chosen answer would be the wrong answer.For the present study, I reanaly...

show abstract

“…It is remarkable because the two alternatives are so salient. On the one hand, there are circumstances in which new information increases confidence more than accuracy, exacerbating overconfidence (Hall, Ariss, & Todorov, 2007;Heath & Gonzalez, 1995;Oskamp, 1965). On the other hand, there are other circumstances in which increasing expertise can increase accuracy faster than confidence, reducing overconfidence (Koriat, Sheffer, & Ma'ayan, 2002).…”

Section: Confidence and Accuracy 23mentioning

confidence: 99%

Confidence Calibration in a Multiyear Geopolitical Forecasting Competition

et al. 2017

View full text Add to dashboard Cite

This research examines the development of confidence and accuracy over time in the context of forecasting. Although overconfidence has been studied in many contexts, little research examines its progression over long periods of time or in consequential policy domains. This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events. We sought to apply insights from research to structure the questions, interactions, and elicitations to improve forecasts. Indeed, forecasters’ confidence roughly matched their accuracy. As information came in, accuracy increased. Confidence increased at approximately the same rate as accuracy, and good calibration persisted. Nevertheless, there was evidence of a small amount of overconfidence (3%), especially on the most confident forecasts. Training helped reduce overconfidence, and team collaboration improved forecast accuracy. Together, teams and training reduced overconfidence to 1%. Our results provide reason for tempered optimism regarding confidence calibration and its development over time in consequential field contexts. This paper was accepted by Yuval Rottenstreich, judgment and decision making.

show abstract

The illusion of knowledge: When more information reduces accuracy and increases confidence

Cited by 168 publications

References 32 publications

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

When confidence in a choice is independent of which choice is made

Confidence Calibration in a Multiyear Geopolitical Forecasting Competition

Contact Info

Product

Resources

About