Uncertain Judgements: Eliciting Experts' Probabilities

Abstract

Statistics in Practice is an important international series of texts, which provide detailed coverage of statistical concepts, methods and worked case studies in specific fields of investigation and study.With sound motivation and many worked practical examples, the books show in down-to-earth terms how to select and use an appropriate range of statistical techniques in a particular practical field within each title's special topic area.The books provide statistical support for professionals and research workers across a range of employment fields and research environments. Subject areas covered include medicine and pharmaceutics; industry, finance and commerce; public services; the earth and environmental sciences, and so on.The books also provide support to students studying statistical courses applied to the above areas. The demand for graduates to be equipped for the work environment has led to such courses becoming increasingly prevalent at universities and colleges.It is our aim to present judiciously chosen and well-written workbooks to meet everyday practical needs. The feedback of views from readers will be most valuable to monitor the success of this aim.A complete list of titles in this series appears at the end of the volume.ii

Policy advice: Use experts wisely

“…Happily, a large and growing body of literature describes methods for engaging with experts that enhance the accuracy and calibration of their judgements 4,5 . Unhappily, these methods are rarely used to support public-policy decisions.…”

Methods to Elicit Probability Distributions from Experts: A Systematic Review of Reported Practice in Health Technology Assessment

“…consensus panels) or mathematical methods (e.g. linear opinion pooling) [16]; -face-to-face facilitated vs. self-administered elicitation -a face-to-face facilitated session requires a trained analyst to guide the experts through the questions, a process that is resource intensive (considering also time and travel-related expenses), while a self-administered questionnaire is easier to send out, but requires careful preparation of the questionnaire, and response rates are typically low [4]; -calibration vs. equal weighting for mathematical synthesis -through calibration, the opinion of more knowledgeable experts can be better represented, but it is difficult to identify relevant weighting criteria (which experts are better informed than others) [23]; questioning strategy -framing bias -the way questions are formulated may influence the estimates [38].…”

“…Ideally, elicitation should be conducted with a number of experts that have the most expertise while not sharing the same perspective [8]. Availability and willingness to participate, as well as potential conflicts of interest also need to be taken into account [4]. There is little literature on the number of experts to elicit from, but research in other fields [9] suggest that between six and twelve experts should be included in most elicitation exercises.…”

“…The question then is whether to elicit the values that represent the location and spread of these distributions directly or indirectly [11]. O'Hagan et al provide a detailed overview of various elicitation methods [4], but only a few have been used by analysts in HTA [12].…”

“…This is a fixed interval method that allows quantities of interest to be elicited graphically. It is a discrete form of the probability density function (PDF) [4], where the expert is presented with a frequency chart on which he or she is asked to place a number of crosses (alternatively called chips or tokens). Placing all the crosses in one column would represent complete certainty, while placing all the crosses on the bottom row would represent complete uncertainty about the true value (Figure 1).…”