An Expert on Every Street Corner? Methods for Eliciting Distributions in Geographically Dispersed Opinion Pools

Abstract

Recent publications outline developments in eliciting probabilistic opinions from clinical experts with which to inform structural assumptions and parameter estimates in health economic models. We outline approaches taken to date to elicit probabilistic distributions from experts within the health economic literature and outline the appropriate considerations and the resulting process in developing a new elicitation program with the aim of allowing low-cost elicitation of expert opinion from a heterogeneous and geographically dispersed opinion pool while preserving the essential features of good practice elicitation methods.

Methods to Elicit Probability Distributions from Experts: A Systematic Review of Reported Practice in Health Technology Assessment

“…Four of the 12 were exclusively dedicated to the description of the elicitation exercise, independent of the modelling process [12,21,26,27], and one [28] was a methodological paper which also reported an elicitation study. The rest of the studies reported the elicitation exercise with less detail, but explicitly indicated elicitation of probability distributions for a number of model parameters.…”

“…Two studies [12,26] report specifically selecting experts with different experiences, while in Stevenson et al [30], the experts were part of the study team. One study [28] specifically targets geographically dispersed experts, in an effort to explore the wide-spread collection of quality data. No further aspects of expert selection are reported in the studies.…”

“…Four studies [12,25,26,29] report constructing pilot exercises to assess various methods to conduct and combine elicitations. Ease of use was an important factor in choosing the appropriate method, as it predicted higher compliance by the experts, especially when a facilitator was not available [25,26,28]. Eleven of the fourteen studies report providing training in probability elicitation for the participating experts prior to undertaking the elicitation session.…”

“…Eleven of the fourteen studies report providing training in probability elicitation for the participating experts prior to undertaking the elicitation session. In seven studies [14,21,[26][27][28][29], experts were given written instructions for completing the questionnaires and in five of these background information on the modelling exercise was given. Four studies reported the presence of a facilitator during the elicitation [12,21,31,32], and in one study [12], experts attended a 2 hour preparatory training session.…”

“…Four studies [12,14,27,29] used the histogram technique and three studies [21,28,30] used the bisection method. Leal et al [26] report the use of a method based on the "six complementary intervals" method (the hybrid method), but using instead only four intervals.…”