Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations

Zondervan‐Zwijnenburg, Mariëlle; Schoot-Hubeek, Wenneke van de; Lek, Kimberley; Hoijtink, Herbert; Schoot, Rens van de

doi:10.3389/fpsyg.2017.00090

Cited by 27 publications

(29 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In Bayesian statistics, there are two sources of information: prior knowledge and data [ 1 , 2 , 3 ]. The prior can be composed of expert knowledge [ 4 , 5 , 6 ]. However, deciding which expert yields the most appropriate information remains a critical challenge, for which we present a solution in this paper.…”

Section: Introductionmentioning

confidence: 99%

Using the Data Agreement Criterion to Rank Experts’ Beliefs

Veen

Stoel²,

Schalken

et al. 2018

Entropy

Self Cite

View full text Add to dashboard Cite

Experts' beliefs embody a present state of knowledge. It is desirable to take this knowledge into account when doing analyses or making decisions. Yet ranking experts based on the merit of their beliefs is a difficult task. In this paper we show how experts can be ranked based on their knowledge and their level of (un)certainty. By letting experts specify their knowledge in the form of a probability distribution we can assess how accurately they can predict new data, and how appropriate their level of (un)certainty is. The expert's specified probability distribution can be seen as a prior in a Bayesian statistical setting. By extending an existing prior-data conflict measure to evaluate multiple priors, i.e. experts' beliefs, we can compare experts with each other and the data to evaluate their appropriateness. Using this method new research questions can be asked and answered, for instance: Which expert predicts the new data best? Is there agreement between my experts and the data? Which experts' representation is more valid or useful? Can we reach convergence between expert judgement and data? We provided an empirical example ranking (regional) directors of a large financial institution based on their predictions of turnover.

show abstract

Section: Introductionmentioning

confidence: 99%

Using the Data Agreement Criterion to Rank Experts’ Beliefs

Veen

Stoel²,

Schalken

et al. 2018

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition to the "check" questions, we compared our elicitation method with the trial roulette method (Gore, 1987; also see Goldstein et al, 2008;Johnson et al, 2010b;Goldstein and Rothschild, 2014;Zondervan-Zwijnenburg et al, 2017); a commonly used elicitation method. In our version of this method, teachers were asked to place digital coins on the scale with 25 "puppets, " for two of their pupils.…”

Section: Construct Validitymentioning

confidence: 99%

Development and Evaluation of a Digital Expert Elicitation Method Aimed at Fostering Elementary School Teachers' Diagnostic Competence

Lek

Schoot

2018

Front. Educ.

Self Cite

View full text Add to dashboard Cite

Expert elicitation-an approach to systematically consult experts and quantify their insights-has been succesfully applied in fields as risk assessment, health and environmental research. Unfortunately, it has never been used within the Educational sciences, while it offers ample opportunities for educational practice, especially when used to foster the accuracy of teacher judgments; generally referred to as their "diagnostic competence." The current paper is the first to explore expert elicitation in an educational context and has two major goals. The first goal is to develop a digital expert elicitation method suitable to be used by elementary school teachers for self-reflection purposes. The second goal is to extensively test the expert elicitation method, using a test panel of 24 primary school teachers for 503 pupils in total. Results regarding the development of the elicitation method and its reliability, construct validity, face validity and feasibility are discussed as well as ideas how this elicitation method can be a valuable self-reflection instrument for teachers. The results are promising: all measures of reliability, feasibility, face validity and construct validity show positive results and teachers are enthusiastic about the possibilities of the method.

show abstract

“…In the first procedural option, we used the procedure of the trial roulette where 20 digital “chips,” starting with the expected minimal and maximum value, each representing five percent of a distribution, were to be placed in a grid following the procedure described by Zondervan-Zwijnenburg et al (2017b) . After placing 20 “chips” the respondents could submit their input and they were provided with visual feedback on the distribution that was fitted based on these 20 “chips.” They could accept the representation or adjust their input.…”

Section: Elicitation Studiesmentioning

confidence: 99%

“…Based on the input provided by the expert, a probability distribution is fitted. The trial roulette method has been validated by Johnson et al (2010b) and Zondervan-Zwijnenburg et al (2017b) in a face-to-face setting.…”

Section: Introductionmentioning

confidence: 99%

“…Moreover, there is no direct research into how accurate experts can assess properties like the mean, mode, or variance for the distribution of an uncertain parameter. Research by Johnson et al (2010b) and Zondervan-Zwijnenburg et al (2017b) provide promising results concerning the trial roulette method. Yet, directly obtaining distributional representations may be inappropriate given experts’ unsatisfactory performance on specifying elements of this distribution.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Proposal for a Five-Step Method to Elicit Expert Judgment

Veen

Stoel²,

Zondervan‐Zwijnenburg

et al. 2017

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Elicitation is a commonly used tool to extract viable information from experts. The information that is held by the expert is extracted and a probabilistic representation of this knowledge is constructed. A promising avenue in psychological research is to incorporated experts’ prior knowledge in the statistical analysis. Systematic reviews on elicitation literature however suggest that it might be inappropriate to directly obtain distributional representations from experts. The literature qualifies experts’ performance on estimating elements of a distribution as unsatisfactory, thus reliably specifying the essential elements of the parameters of interest in one elicitation step seems implausible. Providing feedback within the elicitation process can enhance the quality of the elicitation and interactive software can be used to facilitate the feedback. Therefore, we propose to decompose the elicitation procedure into smaller steps with adjustable outcomes. We represent the tacit knowledge of experts as a location parameter and their uncertainty concerning this knowledge by a scale and shape parameter. Using a feedback procedure, experts can accept the representation of their beliefs or adjust their input. We propose a Five-Step Method which consists of (1) Eliciting the location parameter using the trial roulette method. (2) Provide feedback on the location parameter and ask for confirmation or adjustment. (3) Elicit the scale and shape parameter. (4) Provide feedback on the scale and shape parameter and ask for confirmation or adjustment. (5) Use the elicited and calibrated probability distribution in a statistical analysis and update it with data or to compute a prior-data conflict within a Bayesian framework. User feasibility and internal validity for the Five-Step Method are investigated using three elicitation studies.

show abstract

Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations

Cited by 27 publications

References 33 publications

Using the Data Agreement Criterion to Rank Experts’ Beliefs

Using the Data Agreement Criterion to Rank Experts’ Beliefs

Development and Evaluation of a Digital Expert Elicitation Method Aimed at Fostering Elementary School Teachers' Diagnostic Competence

Proposal for a Five-Step Method to Elicit Expert Judgment

Contact Info

Product

Resources

About