A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis

Kretz, Donald R.; Simpson, Bryce; Graham, Col

doi:10.1109/ths.2012.6459889

Cited by 13 publications

(10 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kretz et al (2012) and Kretz and Granderson (2013) compared ACH with two techniques whose primary function is not hypothesis testing (see Dhami, Belton, & Careless, 2016). Furthermore, some of the studies using a control group were confounded by the fact that some "control" participants were familiar with ACH and may have used it (Kretz et al, 2012;Kretz & Granderson, 2013;Lehner et al, 2008). Finally, none of the studies measured whether analysts in the ACH group applied ACH correctly.…”

Section: Past Research On the Analyses Of Competing Hypothesesmentioning

confidence: 99%

“…The small body of past research is conceptually vague in terms of the features of ACH being tested, although there is a general focus on measuring some aspects of confirmation bias (Convertino, Billman, Pirolli, Massar, & Shrager, 2008;Kretz & Granderson, 2013;Kretz, Simpson, & Graham, 2012;Lehner et al, 2008). Specifically, the studies induce confirmation bias in participants before testing ACH by presenting evidence in stages such that it initially favours one hypothesis, and then in later stages, it either balances out across the hypotheses (Convertino et al, 2008), supports a hypothesis it initially conflicted with (Lehner et al, 2008), or conflicts with the hypothesis it initially supported (Kretz et al, 2012;Kretz & Granderson, 2013).…”

Section: Past Research On the Analyses Of Competing Hypothesesmentioning

confidence: 99%

See 1 more Smart Citation

The “analysis of competing hypotheses” in intelligence analysis

Dhami

Belton

Mandel

2019

Applied Cognitive Psychology

View full text Add to dashboard Cite

SummaryThe intelligence community uses “structured analytic techniques” to help analysts think critically and avoid cognitive bias. However, little evidence exists of how techniques are applied and whether they are effective. We examined the use of the analysis of competing hypotheses (ACH)—a technique designed to reduce “confirmation bias.” Fifty intelligence analysts were randomly assigned to use ACH or not when completing a hypothesis testing task that had probabilistic ground truth. Data on analysts' judgement processes and conclusions were collected using written protocols that were then coded for statistical analyses. We found that ACH‐trained analysts did not follow all of the steps of ACH. There was mixed evidence for ACH's ability to reduce confirmation bias, and we observed that ACH may increase judgement inconsistency and error. It may be prudent for the intelligence community to consider the conditions under which ACH would prove useful and to explore alternatives.

show abstract

Section: Past Research On the Analyses Of Competing Hypothesesmentioning

confidence: 99%

Section: Past Research On the Analyses Of Competing Hypothesesmentioning

confidence: 99%

The “analysis of competing hypotheses” in intelligence analysis

Dhami

Belton

Mandel

2019

Applied Cognitive Psychology

View full text Add to dashboard Cite

show abstract

“…These methods could be leveraged by a system to automatically suggest alternative visualisations or items that have not been utilised. The latter may be useful in mitigating some selection based cognitive biases such as confirmation bias [46]. Another approach to derive human trust measures is to analyse user generated contents.…”

Section: G6: Support the Analyst In Uncertainty Aware Sensemakingmentioning

confidence: 99%

The Role of Uncertainty, Awareness, and Trust in Visual Analytics

Sacha

Senaratne

Kwon

et al. 2016

IEEE Trans. Visual. Comput. Graphics

230

130

View full text Add to dashboard Cite

Visual analytics supports humans in generating knowledge from large and often complex datasets. Evidence is collected, collated and cross-linked with our existing knowledge. In the process, a myriad of analytical and visualisation techniques are employed to generate a visual representation of the data. These often introduce their own uncertainties, in addition to the ones inherent in the data, and these propagated and compounded uncertainties can result in impaired decision making. The user's confidence or trust in the results depends on the extent of user's awareness of the underlying uncertainties generated on the system side. This paper unpacks the uncertainties that propagate through visual analytics systems, illustrates how human's perceptual and cognitive biases influence the user's awareness of such uncertainties, and how this affects the user's trust building. The knowledge generation model for visual analytics is used to provide a terminology and framework to discuss the consequences of these aspects in knowledge construction and though examples, machine uncertainty is compared to human trust measures with provenance. Furthermore, guidelines for the design of uncertainty-aware systems are presented that can aid the user in better decision making.

show abstract

“…Unfortunately, there is little scientific research on ACH, and what exists must be interpreted cautiously for several reasons, such as small sample sizes (e.g., Convertino, Billman, Pirolli, Massar & Shrager, 2008;Lehner, Adelman, Cheikes & Brown, 2008;Kretz, Simpson & Graham, 2012), lack of control groups (Convertino et al, 2008) or appropriate control groups (Kretz et al, 2012). Moreover, virtually all published studies have omitted critical, quantitative measures of judgment accuracy, focusing instead on distal considerations such as whether ACH reduces (the highly equivocal notion of) "confirmation bias" (Nickerson, 1998).…”

Section: The Analysis Of Competing Hypotheses Techniquementioning

confidence: 99%

Boosting intelligence analysts’ judgment accuracy: What works, what fails?

Mandel

Karvetski²,

Dhami

2018

Judgm. decis. mak.

View full text Add to dashboard Cite

A routine part of intelligence analysis is judging the probability of alternative hypotheses given available evidence. Intelligence organizations advise analysts to use intelligence-tradecraft methods such as Analysis of Competing Hypotheses (ACH) to improve judgment, but such methods have not been rigorously tested. We compared the evidence evaluation and judgment accuracy of a group of intelligence analysts who were recently trained in ACH and then used it on a probability judgment task to another group of analysts from the same cohort that were neither trained in ACH nor asked to use any specific method. Although the ACH group assessed information usefulness better than the control group, the control group was a little more accurate (and coherent) than the ACH group. Both groups, however, exhibited suboptimal judgment and were susceptible to unpacking effects. Although ACH failed to improve accuracy, we found that recalibration and aggregation methods substantially improved accuracy. Specifically, mean absolute error (MAE) in analysts’ probability judgments decreased by 61% after first coherentizing their judgments (a process that ensures judgments respect the unitarity axiom) and then aggregating their judgments. The findings cast doubt on the efficacy of ACH, and show the promise of statistical methods for boosting judgment quality in intelligence and other organizations that routinely produce expert judgments.

show abstract

A game-based experimental protocol for identifying and overcoming judgment biases in forensic decision analysis

Cited by 13 publications

References 4 publications

The “analysis of competing hypotheses” in intelligence analysis

The “analysis of competing hypotheses” in intelligence analysis

The Role of Uncertainty, Awareness, and Trust in Visual Analytics

Boosting intelligence analysts’ judgment accuracy: What works, what fails?

Contact Info

Product

Resources

About