Reference Case Methods for Expert Elicitation in Health Care Decision Making

Abstract

Background The evidence used to inform health care decision making (HCDM) is typically uncertain. In these situations, the experience of experts is essential to help decision makers reach a decision. Structured expert elicitation (referred to as elicitation) is a quantitative process to capture experts’ beliefs. There is heterogeneity in the existing elicitation methodology used in HCDM, and it is not clear if existing guidelines are appropriate for use in this context. In this article, we seek to establish reference case methods for elicitation to inform HCDM. Methods We collated the methods available for elicitation using reviews and critique. In addition, we conducted controlled experiments to test the accuracy of alternative methods. We determined the suitability of the methods choices for use in HCDM according to a predefined set of principles for elicitation in HCDM, which we have also generated. We determined reference case methods for elicitation in HCDM for health technology assessment (HTA). Results In almost all methods choices available for elicitation, we found a lack of empirical evidence supporting recommendations. Despite this, it is possible to define reference case methods for HTA. The reference methods include a focus on gathering experts with substantive knowledge of the quantities being elicited as opposed to those trained in probability and statistics, eliciting quantities that the expert might observe directly, and individual elicitation of beliefs, rather than solely consensus methods. It is likely that there are additional considerations for decision makers in health care outside of HTA. Conclusions The reference case developed here allows the use of different methods, depending on the decision-making setting. Further applied examples of elicitation methods would be useful. Experimental evidence comparing methods should be generated.

The Use of Expert Elicitation among Computational Modeling Studies in Health Research: A Systematic Review

“…Journals, particularly those that publish high-quality model-based research, should open venues to facilitate the publication and peer review of stand-alone EEs with emphasis on adhering to principles and reporting guidelines for EEs. 9,18 Separate peer-reviewed publication of EEs should be encouraged, as this would enable independent review of the EE and the subsequent modeling study.…”

“…79 The lack of consensus and guidance on methods has likely contributed to the limited reporting found in this and previous reviews. Earlier this year, Bojke et al 18 sought to develop reference case methods for EEs in the health care setting, offering 9 principles to guide the use of EE. However, modelers are still left with considerable discretion to determine when and how expert opinion should be elicited, aggregated, modeled, and communicated, and readers have almost no ability to evaluate the quality of the resulting parameters and results if editors and reviewers do not know of these guiding principles.…”

“…Despite the widespread use of expert opinion as a source of data for health analyses, no consensus or gold standard exists on the proper methods of planning, implementing, presenting, and using data from an EE. 9,18 There exists a range of formal to informal EE methods. 10,11,[18][19][20][21][22][23][24] The levels of complexity and rigor of the methods vary.…”

“…9,18 There exists a range of formal to informal EE methods. 10,11,[18][19][20][21][22][23][24] The levels of complexity and rigor of the methods vary. For instance, elicitations can be conducted in person, over the phone, online via video conferences, or with paper or electronic surveys.…”

“…10,11 To that end, Iglesias and colleagues 9 developed guidelines for reporting results from EEs. More recently still, Bojke et al 18 developed a set of guiding principles and criticisms to foster the development of a reference case for health-related EEs.…”