Elicitation

Abstract

I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science, whatever the matter may be.

Calibrating Expert Assessments Using Hierarchical Gaussian Process Models

“…The authors provided the experts with data on catch history, limited entries of commercial length compositions, life history and the starting year of management actions, aiming to imitate the amount of information available to experts in real data-limited situations. With the data provided and their prior knowledge attributed to their education and experience, the experts formulated their beliefs about the stock status system parameters as beta distributions with the aid of quantiles, which are typically easier for experts to estimate than means or variances (Garthwaite et al, 2005;O'Hagan et al, 2006;Dias et al, 2018). Here, we will compare the elicited distributions to (point) estimates of the stock status from the corresponding data-rich fisheries stock assessment models or simulated stocks.…”

“…Expert elicitation is an important part of statistical analyses in various fields of research and decision making (O'Hagan et al, 2006;Dias et al, 2018;Albert et al, 2012). In a typical situation where expert knowledge is used, data are lacking or the time and resources to collect the data are limited (Burgman et al, 2011;Morgan, 2014) and thus, the available information is insufficient to make meaningful inference about the phenomenon of interest (Burgman, 2005;Roman et al, 2008;Zickfeld et al, 2010;Wilson et al, 2018).…”

“…The second question concerns the proper usage of the assessments in statistical decision making. How the assessments of multiple experts should be utilized in statistical inference and decision making so that the uncertainties and possible systematic errors or biases in the assessments are properly accounted for (Tversky and Kahneman, 1974;Lindley et al, 1979;O'Hagan et al, 2006;Burgman et al, 2011;Dias et al, 2018)? Even though these issues are intertwined, here we focus on the latter paying special attention to considering the biases, or in other words, to the calibration of experts' assessments.…”

“…Methods for expert elicitation have been extensively studied, and comprehensive reviews and detailed treatments of the subject are provided by O'Hagan et al (2006) and Dias et al (2018). Here, we assume that during the elicitation process, each expert's knowledge has been formulated as a probability distribution.…”

“…The information provided by the experts is linked to the unknown system parameters through a conditional probability distribution (likelihood function). However, it can be challenging to formulate the likelihood function (Genest and Schervish, 1985;O'Hagan et al, 2006), which is probably why the latter approach, the opinion pooling, has gained more popularity (Dias et al, 2018). Opinion pooling, which also forms the basis of the classical models (Cooke and Goossens, 2008), is based on combining the probability distributions provided by the experts either by weighted arithmetic or logarithmic averaging (McConway, 1981;O'Hagan et al, 2006;Dietrich and List, 2014;Farr et al, 2018).…”