SHELF: The Sheffield Elicitation Framework

Gosling, John Paul

doi:10.1007/978-3-319-65052-4_4

Cited by 111 publications

(154 citation statements)

References 40 publications

Supporting

Mentioning

154

Contrasting

Order By: Relevance

“…If the intent is simply to be transparent about uncertainty, then qualitative descriptions may be enough. However, often probability distributions are required when the values of specific variables have to be estimated and in these cases uncertainty has to be quantified, which can be supported by a variety of sophisticated software tools (e.g., EXCALIBUR: Cooke & Solomatine, ; MATCH: Morris, Oakley, & Crowe, ; SHELF: Gosling, ).…”

Section: Discussionmentioning

confidence: 99%

A systematic review of transparency in the methods of expert knowledge use

Drescher

Edwards

2018

Journal of Applied Ecology

View full text Add to dashboard Cite

1. The use of expert knowledge (EK) as an alternative to empirical data is increasing in the ecological disciplines in response to the growing need for rapid decisions that are embraced by local communities and implemented under strong resource constraints. Despite this increasing use, the validity of EK as a data source is still questioned by some who label it as biased or unreliable. Transparency in the methods applied in the use of EK allows confirmation of methodological rigour and study repeatability; key steps towards promoting the acceptance of EK as a valid data source. However, the practiced levels of methodological transparency in EK use are currently unknown in the ecological disciplines.2. To fill this knowledge gap, we performed a systematic review of transparency in methods applied in the use of EK, focusing on research in forest ecology and management as a case study. Covering the period from 1990 to 2016, we identified 296 publications that employed EK, which were assessed for their reporting on 15 methodological criteria of EK use. 3.Our results indicate that transparency in the use of EK is generally very low, with only three out of 15 methodological criteria addressed in more than 50% of publications. Nevertheless, overall reporting on EK methods, and the reporting of four individual methodological criteria, were positively affected by publicationyear, suggesting that researchers increasingly recognize the importance of transparency in the use of EK methods. Synthesis and applications.Given the growing use of expert knowledge (EK) in the ecological disciplines, there is an urgent need to further its acceptance as a valid data source. To achieve this goal, researchers need to be more diligent in following best practices and reporting on EK methods. We therefore present a proposed set of guidelines for the reporting of EK methods. However, journal editors and reviewers also can play an important role in reaching this goal by requiring authors to transparently report on EK methods, just as they would expect for the use of empirical data. K E Y W O R D Selicitation, expert judgement, expert knowledge, forest, methodology, reliability, reporting methods, transparency

show abstract

Section: Discussionmentioning

confidence: 99%

A systematic review of transparency in the methods of expert knowledge use

Drescher

Edwards

2018

Journal of Applied Ecology

View full text Add to dashboard Cite

show abstract

“…One way is through (small‐scale) software applications that automatically show (and visualize) for an expert how much probability mass has been used and is still available to use in a different part of the distribution. This is similar to expert judgment methods that make use of chips when assessing marginal distributions, like placing bets on a roulette table (Gosling, ).…”

Section: Discussionmentioning

confidence: 96%

“…Otherwise, the experts condition their judgments on different marginal probabilities and their assessments cannot be sensibly aggregated. The specification is done either through historical data (if available) or another, prior elicitation with a SEJ method for univariate uncertainty, such as Gosling (), Hanea, Burgman, and Hemming (), and Quigley, Colson, Aspinall, and Cooke (). A structured elicitation for the marginal distributions is also encouraged when eliciting dependence only from one expert, i.e., without aggregation, as this mitigates potential biases of the marginals and ensures transparency (Werner et al., ).…”

Section: An Illustrative Case Study: Assessing Spatial Dependence Of mentioning

confidence: 99%

Sequential Refined Partitioning for Probabilistic Dependence Assessment

Werner

Bedford²,

Quigley³

2018

Risk Analysis

View full text Add to dashboard Cite

Modeling dependence probabilistically is crucial for many applications in risk assessment and decision making under uncertainty. Neglecting dependence between multivariate uncertainties can distort model output and prevent a proper understanding of the overall risk. Whenever relevant data for quantifying and modeling dependence between uncertain variables are lacking, expert judgment might be sought to assess a joint distribution. Key challenges for the use of expert judgment for dependence modeling are over- and underspecification. An expert can provide assessments that are infeasible, i.e., not consistent with any probability distribution (overspecification), and on the other hand, without making very restrictive parametric assumptions an expert cannot fully define a probability distribution (underspecification). The sequential refined partitioning method addresses over- and underspecification while allowing for flexibility about which part of a joint distribution is assessed and its level of detail. Potential overspecification is avoided by ensuring low cognitive complexity for experts through eliciting single conditioning sets and by offering feasible assessment ranges. The feasible range of any (sequential) assessment can be derived by solving a linear programming problem. Underspecification is addressed by modeling the density of directly and indirectly assessed distribution parts as minimally informative given their constraints. Hence, our method allows for modeling the whole distribution feasibly and in accordance with experts' information. A nonparametric way of assessing and modeling dependence flexibly in such detail has not been presented in the expert judgment literature for probabilistic dependence models so far. We provide an example of assessing terrorism risk in insurance underwriting.

show abstract

“…A special pooling rule (referred to as "the classical model") is used in which the experts are weighted according to their performance on the seed variables. • The Sheffield protocol in the EFSA guidance is a particular case of the SHELF protocol (Oakley and O'Hagan 2016;Gosling 2018), and employs behavioral aggregation. It is characterized by two rounds of judgments from the experts.…”

Section: Elicitation Protocolsmentioning

confidence: 99%

Expert Knowledge Elicitation: Subjective but Scientific

O’Hagan

2019

The American Statistician

250

212

View full text Add to dashboard Cite

Expert opinion and judgment enter into the practice of statistical inference and decision-making in numerous ways. Indeed, there is essentially no aspect of scientific investigation in which judgment is not required. Judgment is necessarily subjective, but should be made as carefully, as objectively, and as scientifically as possible. Elicitation of expert knowledge concerning an uncertain quantity expresses that knowledge in the form of a (subjective) probability distribution for the quantity. Such distributions play an important role in statistical inference (for example as prior distributions in a Bayesian analysis) and in evidence-based decision-making (for example as expressions of uncertainty regarding inputs to a decision model). This article sets out a number of practices through which elicitation can be made as rigorous and scientific as possible. One such practice is to follow a recognized protocol that is designed to address and minimize the cognitive biases that experts are prone to when making probabilistic judgments. We review the leading protocols in the field, and contrast their different approaches to dealing with these biases through the medium of a detailed case study employing the SHELF protocol. The article ends with discussion of how to elicit a joint probability distribution for multiple uncertain quantities, which is a challenge for all the leading protocols. Supplementary materials for this article are available online.

show abstract

SHELF: The Sheffield Elicitation Framework

Cited by 111 publications

References 40 publications

A systematic review of transparency in the methods of expert knowledge use

A systematic review of transparency in the methods of expert knowledge use

Sequential Refined Partitioning for Probabilistic Dependence Assessment

Expert Knowledge Elicitation: Subjective but Scientific

Contact Info

Product

Resources

About