Testing Best Practices to Reduce the Overconfidence Bias in Multi-criteria Decision Analysis

Ferretti, Valentina; Güney, Şule; Montibeller, Gilberto; Winterfeldt, Detlof von

doi:10.1109/hicss.2016.195

Cited by 14 publications

(12 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results show a clear benefit of the MOLE technique for both the calibration and accuracy of elicited ranges. We found little support, however, for the role of initial best guesses or simple counter-intuitive values in improving elicitationsthe latter observation being in line with Ferretti et al's (2016) results.…”

Section: Discussionsupporting

confidence: 48%

“…-that can be answered using natural, frequentist reasoning. In fact, while there is evidence that people are better at evaluating than generating ranges (Winman, Hansson, & Juslin, 2004), range evaluation still results in some overconfidence (Winman et al, 2004) and recent work by Ferretti, Guney, Montibeller and von Winterfeldt (2016), found limited benefit in an experiment where participants both generated and then evaluated a range. Given this, overconfidence can not be dismissed.…”

Section: Overconfidencementioning

confidence: 99%

“…For instance, there is evidence that splitting the task into partsasking for the 10 th and 90 th percentiles separately rather than for a range that a person is 80% confident will contain the true valueproduces better results, possibly as a result of lifting limitations on cognitive effort (by splitting a single task into two tasks) (see, e.g., Juslin, Wennerholm, & Olsson, 1999). Other debiasing techniques, proposed in Montbellier and von Winterfeldt's (2015) and tested in Ferretti et al (2016) include the use of bets to identify errors in probability and the presentation of counterfactuals (values lying outside the initial range). Both were found to have only small effects on the width of elicited ranges although the use of bets improved the best estimate.…”

Section: Overconfidencementioning

confidence: 99%

See 2 more Smart Citations

More-or-less elicitation (MOLE): reducing bias in range estimation and forecasting

Welsh

Begg

2018

EURO Journal on Decision Processes

View full text Add to dashboard Cite

Self-archiving for articles in subscription-based journalsSpringer journals' policy on preprint sharing.By signing the Copyright Transfer Statement you still retain substantial rights, such as self-archiving: Author(s) are permitted to self-archive a pre-print and an author's accepted manuscript version of their Article. ……….b. An Author's Accepted Manuscript (AAM) is the version accepted for publication in a journal following peer review but prior to copyediting and typesetting that can be made available under the following conditions:(i) Author(s) retain the right to make an AAM of their Article available on their own personal, selfmaintained website immediately on acceptance, (ii) Author(s) retain the right to make an AAM of their Article available for public release on any of the following 12 months after first publication ("Embargo Period"): their employer's internal website; their institutional and/or funder repositories. AAMs may also be deposited in such repositories immediately on acceptance, provided that they are not made publicly available until after the Embargo Period.

show abstract

Section: Discussionsupporting

confidence: 48%

Section: Overconfidencementioning

confidence: 99%

Section: Overconfidencementioning

confidence: 99%

See 1 more Smart Citation

More-or-less elicitation (MOLE): reducing bias in range estimation and forecasting

Welsh

Begg

2018

EURO Journal on Decision Processes

View full text Add to dashboard Cite

show abstract

“…local temperature and the Dow Jones), Abbas et al 160 found less evidence of overconfidence using the fixed-value method. However, Ferretti et al 148 found that this resulted in relatively little improvement in performance. Hence, although there is some evidence that fixed-value approaches may reduce overconfidence, this is limited.…”

Section: Fixed Value Compared With Fixed Probability Methodsmentioning

confidence: 99%

“…This is consistent with research suggesting that people may be better at evaluating confidence intervals than providing them. 146,147 More recently, Ferreti et al 148 noted reductions in overconfidence when environmental science students were instructed to (1) actively think of reasons why their initial highest and lowest estimates of sea level rise may be incorrect; and (2) consider their willingness to place hypothetical bets on elicited confidence intervals.…”

Section: Consider More Informationmentioning

confidence: 99%

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

Bojke

Soares

Claxton

et al. 2021

Health Technol Assess

View full text Add to dashboard Cite

Background Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making. Objectives The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts’ judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed. Methods A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the ‘choices’) and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken. Results For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution. Limitations The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are. Conclusions The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required. Funding This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).

show abstract

Exploring fishing threat at fleet segment and subregional scale: Least expert knowledge and a resilience versus disturbance‐based approach as conservation's tools for cartilaginous fish

Scacco

Crescenzo

Sbrana

2023

Ecology and Evolution

View full text Add to dashboard Cite

show abstract

Testing Best Practices to Reduce the Overconfidence Bias in Multi-criteria Decision Analysis

Abstract: Abstract

Cited by 14 publications

References 57 publications

More-or-less elicitation (MOLE): reducing bias in range estimation and forecasting

More-or-less elicitation (MOLE): reducing bias in range estimation and forecasting

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

Exploring fishing threat at fleet segment and subregional scale: Least expert knowledge and a resilience versus disturbance‐based approach as conservation's tools for cartilaginous fish

Contact Info

Product

Resources

About