Boosting intelligence analysts’ judgment accuracy: What works, what fails?

Abstract

A routine part of intelligence analysis is judging the probability of alternative hypotheses given available evidence. Intelligence organizations advise analysts to use intelligence-tradecraft methods such as Analysis of Competing Hypotheses (ACH) to improve judgment, but such methods have not been rigorously tested. We compared the evidence evaluation and judgment accuracy of a group of intelligence analysts who were recently trained in ACH and then used it on a probability judgment task to another group of analysts from the same cohort that were neither trained in ACH nor asked to use any specific method. Although the ACH group assessed information usefulness better than the control group, the control group was a little more accurate (and coherent) than the ACH group. Both groups, however, exhibited suboptimal judgment and were susceptible to unpacking effects. Although ACH failed to improve accuracy, we found that recalibration and aggregation methods substantially improved accuracy. Specifically, mean absolute error (MAE) in analysts’ probability judgments decreased by 61% after first coherentizing their judgments (a process that ensures judgments respect the unitarity axiom) and then aggregating their judgments. The findings cast doubt on the efficacy of ACH, and show the promise of statistical methods for boosting judgment quality in intelligence and other organizations that routinely produce expert judgments.

A universal method for evaluating the quality of aggregators

“…Aggregation can benefit from transforming the original judgements, using some predefined function, and aggregating the transformed estimates. Various transformation methods have been proved to enhance the aggregated forecasting accuracy (Ariely et al 2000;Baron et al, 2014;Satopää et al, 2014;Mandel, Karvetski & Dhami, 2018;Turner, Steyvers, Merkle, Budescu & Wallsten, 2014). The quantile metric can also be used to evaluate the effectiveness of these transformations.…”

Coherence of probability judgments from uncertain evidence: Does ACH help?

“…Whitesmith (2019) found that ACH did not mitigate either confirmation bias or serial-position effects on the interpretation of evidence in reasoning about an intelligence analysis scenario (also see Lehner, Adelman, Cheikes & Brown, 2008). Mandel, Karvetski and Dhami (2018) found that intelligence analysts who were instructed to use ACH on a probabilistic hypothesis-testing task were less coherent and marginally less accurate in assigning probabilities to the hypotheses than analysts in a control group who were not instructed to use any SAT. Moreover, a subsequent examination of analysts' information use from the same experiment showed that analysts in the ACH condition were less likely to use relevant base-rate information than analysts in the control condition (Dhami, Belton & Mandel, 2019).…”

Recalibrating probabilistic forecasts to improve their accuracy

“…Various transformation methods have been developed and proved to enhance the forecasting accuracy (Ariely et al, 2000;Baron et al, 2014;Satopää et al, 2014;Mandel, Karvetski & Dhami, 2018). Turner et al (2014) discussed the Lin-We use the terms judge, forecaster and expert interchangeably throughout the paper.…”