Experimental Investigation on the Elicitation of Subjective Distributions

Abstract

Elicitation methods aim to build participants' distributions about a parameter of interest. In most elicitation studies this parameter is rarely known in advance and hinders an objective comparison between elicitation methods. In two experiments, participants were first presented with a fixed random sequence of images and numbers and subsequently their subjective distributions of percentages of one of those numbers was elicited. Importantly, the true percentage was set in advance. The first experiment tested whether receiving instructions as to the elicitation method would assist in estimating a true value more accurately than receiving no instructions and whether accuracy was determined by the numerical skills of the participants. The second experiment sought to compare the elicitation method used in the first experiment with a variation of a graphical elicitation method. The results indicate that (i) receiving instructions as to the elicitation method does assist in producing estimates closer to a true percentage value, (ii) the level of numerical skills does not play a part in the accuracy of the estimation (Experiment 1), and (iii) although the average estimates of the betting and graphical method are not significantly different, the betting method leads to more precise estimations than the graphical method (Experiment 2). Both studies featured statistical procedures (functional data analysis and a novel clustering technique) not considered in past research on the elicitation of subjective distributions. The implications of these results are discussed in relation to a recent key study.

AI-powered narrative building for facilitating public participation and engagement

“…While NLP enables parsing large amounts of text and semantic differential rating scales enable to capture quantitative data, EKE is a technique to extract knowledge from human experts in relation to a topic of interest [ 4 , 65 , 78 ]. Some of the classic EKE techniques are interviews, protocol analysis, concept sorting, among others [ 78 ].…”

“…Some of the classic EKE techniques are interviews, protocol analysis, concept sorting, among others [ 78 ]. Other approaches based on statistics, probability, and decision-making theories, elicit quantitative knowledge by obtaining information about an uncertain quantity from a person’s subjective experience and converting it into a probability distribution [ 4 , 33 , 65 ]. Each EKE approach can be used for categorical (e.g., interviews) or numeric topics (e.g., via subjective probability distributions) but the key feature of EKE is that it allows eliciting judgments from people in a highly systematic and scientific fashion.…”

Effect of knowledge differentiation and state space partitioning on subjective probability estimation

“…The results of the experiment showed that the group with instructions estimated the parameter more accurately than the group without. 11 The author investigates the effect of knowledge on subjective probability estimation in this study. Based on literature review, the second hypothesis is formulated: Hypothesis 2 (H2): The level of knowledge provided to the participants affects subjective probability estimation.…”

An FDA-Based Approach for Clustering Elicited Expert Knowledge

“…An elicitation process involves extracting information about parameters of interest (θ) from the subjective experience of a person (expert) and expressing it as a probability distribution (prior distribution) Barrera-Causil et al, (2019) [2]. Thus, for expert i, a real function x i (θ) (prior distribution) is elicited, and for a grid of m equally spaced points…”

“…Hence, we believe that the probability distributions of each expert can be classified and their opinions represented using clusters of beliefs. Thus, Bayesian inference can be carried out in parallel by considering each cluster of priors and a decision can be arrived at via experts' criteria (Barrera-Causil et al, 2019) [2]. On the other hand, classifying probability distributions is an essential task in different areas.…”