Individuals vs. BARD: Experimental Evaluation of an Online System for Structured, Collaborative Bayesian Reasoning

Abstract

US intelligence analysts must weigh up relevant evidence to assess the probability of their conclusions, and express this reasoning clearly in written reports for decision-makers. Typically, they work alone with no special analytic tools, and sometimes succumb to common probabilistic and causal reasoning errors. So, the US government funded a major research program (CREATE) for four large academic teams to develop new structured, collaborative, software-based methods that might achieve better results. Our team's method (BARD) is the first to combine two key techniques: constructing causal Bayesian network models (BNs) to represent analyst knowledge, and small-group collaboration via the Delphi technique. BARD also incorporates compressed, high-quality online training allowing novices to use it, and checklist-inspired report templates with a rudimentary AI tool for generating text explanations from analysts' BNs. In two prior experiments, our team showed BARD's BN-building assists probabilistic reasoning when used by individuals, with a large effect (Glass' 0.8) (Cruz et al., 2020), and even minimal Delphi-style interactions improve the BN structures individuals produce, with medium to very large effects (Glass' 0.5-1.3) (Bolger et al., 2020). This experiment is the critical test of BARD as an integrated system and possible alternative to business-as-usual for intelligence analysis. Participants were asked to solve three probabilistic reasoning problems spread over 5 weeks, developed by our team to test both quantitative accuracy and susceptibility to tempting qualitative fallacies. Our 256 participants were randomly assigned to form 25 teams of 6-9 using BARD and 58 individuals using Google Suite and (if desired) the best pen-and-paper techniques. For each problem, BARD outperformed this control with very large to huge effects (Glass' 1.4-2.2), greatly exceeding CREATE's initial target. We conclude that, for suitable problems, BARD already offers significant advantages over both business-as-usual and existing BN software. Our effect sizes also suggest BARD's BN-building and collaboration combined beneficially and cumulatively, although implementation differences decreased performances compared to Cruz et al. (2020), so interaction may have contributed. BARD has enormous potential for further development and testing of specific components and on more complex problems, and many potential applications beyond intelligence analysis.

BARD: A Structured Technique for Group Elicitation of Bayesian Networks to Support Analytic Reasoning

“…If there are two incompatible but highly rated models or reports, then the facilitator could choose to collate them as two distinct, alternative views in the group's report, e.g., of the majority and minority. 20 BARD includes functionality to officially "submit" the final group solution, which is useful for hu- 19 For an example of an model solution written within (and displaying) BARD's integrated report template, which was annotated and presented to experimental participants in Korb et al (2020) as part of their training, see "Smoking and Cancer -Problem Statement" at https://bit.ly/2V2rwl4 and "Smoking and Cancer -Annotated Solution" at https://bit.ly/3icbBKd. 20 Such rival models could also be exported for prediction via model averaging, which is a common Bayesian technique but not directly supported in BARD.…”

“…In the experimental condition (N = 198), 25 teams consisting of six to nine analysts and a facilitator received BARD training and used the BARD workflow. In the control condition (N = 58), individuals received generic training from the "Guide to Good Reasoning," and specific train-24 For a simple example of this type of problem and rubric, which was used in Korb et al (2020), see "Smoking and Cancer -Problem Statement" at https://bit.ly/2V2rwl4, and "Smoking and Cancer -Rubric" at https://bit.ly/2v92emU. 25 Glass' measures effect size in standard deviations of the control group, which is more appropriate than using Cohen's d with pooled standard deviation of the control and treatment groups when comparing the results of separate experiments on each of several possible treatments, as occurred in CREATE.…”

“…27 In addition, participants from the experimental condition were surveyed at the end of the experiment to capture feedback on BARD usability, us-26 These include a word processor and a spreadsheet for numerical calculations. See https://workspace.google.com/ 27 We summarize the evidence on this issue in Korb et al (2020).…”

“… 19 For an example of an model solution written within (and displaying) BARD's integrated report template, which was annotated and presented to experimental participants in Korb et al. ( 2020 ) as part of their training, see “Smoking and Cancer ‐ Problem Statement” at https://bit.ly/2V2rwl4 and “Smoking and Cancer ‐ Annotated Solution” at https://bit.ly/3icbBKd . …”

“… 24 For a simple example of this type of problem and rubric, which was used in Korb et al. ( 2020 ), see “Smoking and Cancer ‐ Problem Statement” at https://bit.ly/2V2rwl4 , and “Smoking and Cancer ‐ Rubric” at https://bit.ly/2v92emU . …”