Harnessing the wisdom of the inner crowd

Herzog, Stefan M.; Hertwig, Ralph

doi:10.1016/j.tics.2014.06.009

Cited by 70 publications

(61 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Whether and when strategy selection or strategy blending is more accurate is a more complex issue. To appreciate the conditions under which blending strategies improves or decreases accuracy relative to selecting a strategy, it is useful to view blending strategies as applying the concept of the “wisdom of crowds” (Page, ; Surowiecki, ) to a single mind (Herzog & Hertwig, ). Blending estimates or decisions from different people or algorithms, and the conditions under which blending enhances or decreases accuracy, has been discussed in fields ranging from psychology and cognitive science to judgment and decision making, management science, economics, biology, statistics, artificial intelligence, and machine learning (e.g., Armstrong, ; Brown, Wyatt, Harris, & Yao, ; Davis‐Stober, Budescu, Dana, & Broomell, ; Grofman, Owen, & Feld, ; Hastie & Kameda, ; J. Krause, Ruxton, & Krause, ; Kuncheva, ; Larrick & Soll, ; Larrick, Mannes, & Soll, ; Lee, Zhang, & Shi, ; Luan, Katsikopoulos, & Reimer, ; Marling, Sqalli, Rissland, Munoz‐Avila, & Aha, ; Page, ; Timmermann, ).…”

Section: Strategy Selection Versus Strategy Blending: a Normative Permentioning

confidence: 99%

“…Blending estimates or decisions from different people or algorithms, and the conditions under which blending enhances or decreases accuracy, has been discussed in fields ranging from psychology and cognitive science to judgment and decision making, management science, economics, biology, statistics, artificial intelligence, and machine learning (e.g., Armstrong, ; Brown, Wyatt, Harris, & Yao, ; Davis‐Stober, Budescu, Dana, & Broomell, ; Grofman, Owen, & Feld, ; Hastie & Kameda, ; J. Krause, Ruxton, & Krause, ; Kuncheva, ; Larrick & Soll, ; Larrick, Mannes, & Soll, ; Lee, Zhang, & Shi, ; Luan, Katsikopoulos, & Reimer, ; Marling, Sqalli, Rissland, Munoz‐Avila, & Aha, ; Page, ; Timmermann, ). More recently, the study of the wisdom of crowds has been extended to the “inner crowd” (Herzog & Hertwig, ), where several judgments from the same person are blended (Herzog & Hertwig, , ; Vul & Pashler, ).…”

Section: Strategy Selection Versus Strategy Blending: a Normative Permentioning

confidence: 99%

See 1 more Smart Citation

Strategy Selection Versus Strategy Blending: A Predictive Perspective on Single‐ and Multi‐Strategy Accounts in Multiple‐Cue Estimation

Herzog

Helversen

2016

Behavioral Decision Making

Self Cite

View full text Add to dashboard Cite

The claim that a person can use different strategies or processes to solve the same task is pervasive in decision making, categorization, estimation, reasoning, and other research fields. Yet such multi-strategy approaches differ widely in how they envision that the different strategies are coordinated and therefore do not represent one unitary approach. Toolbox models, for example, assume that people shift from one strategy to another as they adapt to specific task environments based on past experience. Unlike such multi-strategy selection approaches, multi-strategy blending approaches assume that the outputs of different strategies are blended into a joint, hybrid response (i.e., "wisdom of strategies" in one mind). The goal of this article is twofold. First, we discuss strategy blending as a conceptual alternative to strategy selection for modeling human judgment. Second, we investigate the predictive performance of the different approaches in synthetic and real-world environments. Taking a normative perspective, we study the coordination of rule-based and exemplar-based processes in estimation tasks. Our simulations using synthetic and real-world environments indicate that, for medium-sized samples, multi-strategy blending approaches lead to more accurate estimates than relying on a single strategy or selecting a strategy based on past experience-possibly because neither rule-nor exemplar-based processes in isolation are sufficient to capture statistical regularities that enable accurate estimates. This suggests that multi-strategy blending approaches can be advantageous to the degree that they rely on qualitatively different strategies. We thank Susannah Goss for editing the manuscript, and the Swiss National Science Foundation for a grant to the first author (100014_129572/1) and a grant to the second author (100014_146169).Correspondence concerning this article should be addressed to Stefan M. Herzog, Center for Adaptive Rationality, Max Planck Institute for Human Development, Lentzeallee 94, 14195 Berlin, Germany. E-mail: herzog@mpib-berlin.mpg.de This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions. STRATEGY SELECTION VS. BLENDING 2 AbstractThe claim that a person can use different strategies or processes to solve the same task is pervasive in decision making, categorization, estimation, reasoning, and other research fields. Yet such multi-strategy approaches differ widely in how they envision that the different strategies are coordinated and therefore do not represent one unitary approach. Toolbox models, for example, assume that people shift from one strategy to another as they adapt to specific task environments based on past experience. Unlike such multi-strategy selection approaches, multi-strategy blending approaches assume that the outputs of different strategies are blended into a joint, hybrid response (i.e., "wisdom of strategies" in one mind).The goal of this article is twofold. First, we introduce strat...

show abstract

Section: Strategy Selection Versus Strategy Blending: a Normative Permentioning

confidence: 99%

Section: Strategy Selection Versus Strategy Blending: a Normative Permentioning

confidence: 99%

Strategy Selection Versus Strategy Blending: A Predictive Perspective on Single‐ and Multi‐Strategy Accounts in Multiple‐Cue Estimation

Herzog

Helversen

2016

Behavioral Decision Making

Self Cite

View full text Add to dashboard Cite

show abstract

“…In a similar way, radiologists could use this approach for low-confidence diagnoses. Our results, combined with the research on the wisdom of the inner crowd [43,44], suggest that it is beneficial to deliberately reconsider low-confidence decisions, and then, if two decisions are inconsistent, to apply the confidence rule.…”

Section: Discussionmentioning

confidence: 66%

“…First, to reduce inconsistency and thus improve accuracy, previous perspectives suggest the use of interventions and methods that improve the overall reliability of information processing, such as reducing the amount of information presented [40], decomposing a complex task into smaller subtasks [41,42], combining an individual's repeated judgments [43][44][45], or requiring individuals to justify their judgments [46,47]. Our perspective suggests a different, but complemen-tary approach to improving accuracy, namely, encouraging experts to make a second assessment whenever their initial decision was low in confidence and to then apply the confidence rule across the two decisions.…”

Section: Discussionmentioning

confidence: 99%

“…Another possibility would be for the software to automatically implement the confidence rule. If a software-driven approach is not feasible, research on the wisdom of the inner crowd [43,44] suggests that prompting individuals to reconsider their initial decision and make a new decision can elicit diverse knowledge that can in turn lead to diverse judgments. Aggregating such diverse judgments can cancel out error, resulting in a more accurate judgment than the initial one.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

How experts' own inconsistency relates to their confidence and between-expert disagreement

Litvinova¹,

Kurvers²,

Hertwig³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Data and code to reproduce the analyses reported in this paper are publicly available at https://osf.io/ chgu5. For more information see the Data availability statement at the end of the paper. 1 the model's predictions in two real-world datasets: diagnosticians rating the same mammograms [19] or images of the lower spine [20] twice. To preview one major insight not anticipated by previous accounts of expert inconsistency: Cases on which there was clear between-expert agreement were associated with highly confident initial decisions that were unlikely to change-independent of whether the experts' consensus decision was correct or incorrect.A model linking experts' inconsistency and confidence to a case's ambiguity A fundamental process assumed by many models of cognition, judgment, and decision making is that individuals sample evidence from their environment or memory when making a decision [12-14, 18, 21]. This sampled evidence determines both the decision and the confidence in that decision [18,21,22]. The assumption is that an individual samples several pieces of evidence ("cues") and selects the option for which there is stronger evidence. The more clearly the evidence points to the option, the more confident the individual will be in the accuracy of their decision. In particular, making a second decision about the same case is seen as equivalent to drawing a second sample of evidence. Because the sampling process is probabilistic, the evidence in the second sample can differ from that in the first sample-as can the corresponding decision (e.g., "cancer" vs. "no cancer") and the individual's confidence in that decision.

show abstract

Wisdom of the inner crowd benefits both face and voice matching

Kramer,

Flanagan,

Jones

et al. 2023

Applied Cognitive Psychology

View full text Add to dashboard Cite

Identification often involves determining whether two face photographs or voice samples originated from the same person. Here, we investigated the wisdom of the (outer) crowd (averaging two individuals' responses to the same trial) and inner crowd (averaging the same individual's responses to the same trial after completing the test twice) as routes to increased performance. Participants completed the same face (Experiment 1) or voice matching test (Experiment 2) twice with no delay. In addition, we reanalysed previously collected data where these tests were completed with a one‐week interval between sessions. For both tests, whether with or without a delay, inner crowds outperformed participants' individual test responses and were equivalent to outer crowds of two participants. Taken together, we demonstrate the use of inner crowds as a robust method of improvement during identification. In contexts where outer crowds are not feasible, agencies should consider inner crowds as a promising alternative.

show abstract

Harnessing the wisdom of the inner crowd

Cited by 70 publications

References 15 publications

Strategy Selection Versus Strategy Blending: A Predictive Perspective on Single‐ and Multi‐Strategy Accounts in Multiple‐Cue Estimation

Strategy Selection Versus Strategy Blending: A Predictive Perspective on Single‐ and Multi‐Strategy Accounts in Multiple‐Cue Estimation

How experts' own inconsistency relates to their confidence and between-expert disagreement

Wisdom of the inner crowd benefits both face and voice matching

Contact Info

Product

Resources

About