Improving the production and evaluation of structural models using a Delphi process

Abstract

Bayes Nets (BNs) are extremely useful for causal and probabilistic modelling in many real-world applications, often built with information elicited from groups of domain experts. But their potential for reasoning and decision support has been limited by two major factors: the need for significant normative knowledge, and the lack of any validated methods or software supporting collaboration. Consequently, we have developed a web-based structured technique – Bayesian Argumentation via Delphi (BARD) – to enable groups of domain experts to receive minimal normative training and then collaborate effectively to produce high-quality BNs. BARD harnesses multiple perspectives on a problem, while minimising biases manifest in freely interacting groups, via a Delphi process: solutions are first produced individually, then shared, followed by an opportunity for individuals to revise their solutions. To test the hypothesis that BNs improve due to Delphi, we conducted an experiment whereby individuals with a little BN training and practice produced structural models using BARD for two Bayesian reasoning problems. Participants then received 6 other structural models for each problem, rated their quality on a 7-point scale, and revised their own models if they wished. Both top-rated and revised models were on average significantly better quality (scored against a gold-standard) than the initial models, with large and medium effect sizes. We conclude that Delphi – and BARD – improves the quality of BNs produced by groups. Further, although rating cannot create new models, rating seems quicker and easier than revision and yielded significantly better models – so, we suggest efficient BN amalgamation should include both.

BARD: A structured technique for group elicitation of Bayesian networks to support analytic reasoning

“…Piloting suggested that for our participants and tasks the trade-off is worthwhile. We also have some experimental evidence that even a minimal Delphi-style interaction improves the network structures produced by BARD groups (Bolger et al, 2019). Nevertheless, we do not yet have any direct experimental comparison for BARD groups between traditional Delphi, realtime Delphi, and free interaction.…”

BARD: A Structured Technique for Group Elicitation of Bayesian Networks to Support Analytic Reasoning

“…Piloting suggested that for our participants and tasks the tradeoff is worthwhile. We also have some experimental evidence that even a minimal Delphi-style interaction improves the network structures produced by BARD groups (Bolger et al, 2020). Nevertheless, we do not yet have any direct experimental compari-29 A beta-version of BARD is now available that supports decision and utility nodes, and computes expected utilities for decisions.…”

“…( 2020 ). This includes a follow‐up experiment we conducted (Bolger et al., 2020 ) to verify whether, and explore how, BARD's Delphi processes contribute, focusing on the core BN‐modeling task of deciding on causal structure. Accuracy of causal structure was measured in a simple, standard way by the edit distance (i.e., the number of arrows that differ) between the structure participants produced and the normatively correct structure.…”

Individuals vs. BARD: Experimental Evaluation of an Online System for Structured, Collaborative Bayesian Reasoning

“…In the Structure Delphi experiment (Bolger et al, 2020), individual participants who had previously used BARD were asked to analyze some of our other reasoning problems they had not previously seen, but only for the critical and most distinctive subtask in BN construction: selecting the right variables and causal structure. All other subtasks in solving our reasoning problems are similar to tasks for which Delphi has already been shown to be effective in prior literature.…”

“…Given that our X participants were supposed to achieve better written reports than K by constructing BN models, it is natural to ask how accurate their models turned out to be, and how well-correlated this was to the quality of their written reports. In Bolger et al (2020), our BARD team members assessed the quality of BN structures by measuring the difference between these and normatively correct "gold-standard" structures using "edit distance, " which is the most well-known structural measure in the literature (e.g., Spirtes et al, 2000). However, this approach was facilitated by requiring participants to choose variables out of a set provided, and not requiring probabilities to be entered in the models, thus avoiding both sources of variation in participant answers.…”