Confidence, advice seeking and changes of mind in decision making

Pescetelli, Niccolò; Hauperich, Anna-Katharina; Yeung, Nick

doi:10.1016/j.cognition.2021.104810

Cited by 38 publications

(35 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They did so by capitalizing on explicit general warnings of ai accuracy but, more importantly, on cues in the ai output, spontaneously interpreted as informative on the ai ’s confidence in its diagnosis: the persistence of the same ai diagnosis, and the rate of no-prediction ai output (Appendix B.4 ). md seemed to evaluate ai confidence as they would have evaluated a colleague’s by perceiving his/her hesitation 44 .…”

Section: Discussionmentioning

confidence: 99%

Experimental evidence of effective human–AI collaboration in medical decision-making

Reverberi

Rigon

Solari

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Artificial Intelligence (ai) systems are precious support for decision-making, with many applications also in the medical domain. The interaction between mds and ai enjoys a renewed interest following the increased possibilities of deep learning devices. However, we still have limited evidence-based knowledge of the context, design, and psychological mechanisms that craft an optimal human–ai collaboration. In this multicentric study, 21 endoscopists reviewed 504 videos of lesions prospectively acquired from real colonoscopies. They were asked to provide an optical diagnosis with and without the assistance of an ai support system. Endoscopists were influenced by ai ($$\textsc {or}=3.05$$ O R = 3.05 ), but not erratically: they followed the ai advice more when it was correct ($$\textsc {or}=3.48$$ O R = 3.48 ) than incorrect ($$\textsc {or}=1.85$$ O R = 1.85 ). Endoscopists achieved this outcome through a weighted integration of their and the ai opinions, considering the case-by-case estimations of the two reliabilities. This Bayesian-like rational behavior allowed the human–ai hybrid team to outperform both agents taken alone. We discuss the features of the human–ai interaction that determined this favorable outcome.

show abstract

Section: Discussionmentioning

confidence: 99%

Experimental evidence of effective human–AI collaboration in medical decision-making

Reverberi

Rigon

Solari

et al. 2022

Sci Rep

View full text Add to dashboard Cite

show abstract

“…Our work builds on previous research into group decisions and information integration processes (Bonaccio & Dalal, 2006; Rader, Larrick, & Soll, 2017; Sniezek & Buckley, 1995; Yaniv & Kleinberger, 2000). It is already well-known that confidence of both judge and advisors act as a weight in opinion aggregation: Confident judges are less likely to ask for advice and are less influenced by advice they receive (Pescetelli, Hauperich, & Yeung, 2020; Tost, Gino, & Larrick, 2012); meanwhile, confident people are trusted more and are more influential within groups and juries (Penrod & Cutler, 1995; Roediger, Roediger, Wixted, & Desoto, 2012; Swol & Sniezek, 2005; Zarnoth & Sniezek, 1997), irrespective of true accuracy (Hertz, Romand-Monnier, Kyriakopoulou, & Bahrami, 2016; Mahmoodi et al, 2015). According to a confidence heuristic (Price & Stone, 2004; Pulford, Colman, Buabang, & Krockow, 2018) an advisor’s confidence signals their likely accuracy.…”

mentioning

confidence: 99%

The role of decision confidence in advice-taking and trust formation.

Pescetelli¹,

Yeung²

2021

Journal of Experimental Psychology: General

Self Cite

View full text Add to dashboard Cite

In a world where ideas flow freely across multiple platforms, people must often rely on others' advice and opinions without an objective standard to judge whether this information is accurate. The present study explores the hypothesis that an individual's internal decision confidence can be used as a signal to learn the accuracy of others' advice, even in the absence of feedback. According to this "agreement-inconfidence" hypothesis, people can learn about an advisor's accuracy across multiple interactions according to whether the advice offered agrees with their own initial opinions, weighted by the confidence with which these initial opinions are held. We test this hypothesis using a judge-advisor system paradigm to precisely manipulate the profiles of virtual advisors in a perceptual decision-making task. We find that when advisors' and participants' judgments are independent, people can correctly learn advisors' features, like their accuracy and calibration, whether or not objective feedback is available. However, when their judgments (and thus errors) are correlated-as is the case in many real social contexts-predictable distortions in trust can be observed between feedback and feedback-free scenarios. Using agent-based simulations, we explore implications of these individual-level heuristics for networklevel patterns of trust and belief formation.

show abstract

“…In our second goal, we use the cognitive modeling approach to understand how a human's reliance policy depends on a number of factors related to the human and the AI. Previous research has shown that a human's confidence in their own decision influences the tendency to rely on AI assistance (Lu & Yin, 2021;Pescetelli et al, 2021;Wang et al, 2022). In addition, reliance on the AI is also affected by the AI's confidence in its decision (Y.…”

Section: Introductionmentioning

confidence: 99%

“…In some previous research, only a single AI model with a fixed degree of accuracy was used; for example, an AI model with an accuracy comparable to human performance (Y. Zhang et al, 2020) or above human performance (Lai & Tan, 2019;Pescetelli et al, 2021). A few studies have investigated the effect of varying AI accuracy on reliance strategy (Yin et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

AI-Assisted Decision-Making: A Cognitive Modeling Approach to Infer Latent Reliance Strategies

Tejeda¹,

Kumar²,

Smyth³

et al. 2022

Preprint

View full text Add to dashboard Cite

AI assistance is readily available to humans in a variety of decision-making applications. In order to fully understand the efficacy of such joint decision-making, it is important to first understand the human's reliance on AI. However, there is a disconnect between how joint decision-making is studied and how it is practiced in the real world. More often than not, researchers ask humans to provide independent decisions before they are shown AI assistance. This is done to make explicit the influence of AI assistance on the human's decision. We develop a cognitive model that allows us to infer the latent reliance strategy of humans on AI assistance without asking the human to make an independent decision. We validate the model's predictions through two behavioral experiments. The first experiment follows a concurrent paradigm where humans are shown AI assistance alongside the decision problem. The second experiment follows a sequential paradigm where humans provide an independent judgment on a decision problem before AI assistance is made available. The model’s predicted reliance strategies closely track the strategies employed by humans in the two experimental paradigms. Our model provides a principled way to infer reliance on AI-assistance and may be used to expand the scope of investigation on human-AI collaboration.

show abstract

Confidence, advice seeking and changes of mind in decision making

Cited by 38 publications

References 51 publications

Experimental evidence of effective human–AI collaboration in medical decision-making

Experimental evidence of effective human–AI collaboration in medical decision-making

The role of decision confidence in advice-taking and trust formation.

AI-Assisted Decision-Making: A Cognitive Modeling Approach to Infer Latent Reliance Strategies

Contact Info

Product

Resources

About