Flexible versus structured support for reasoning: enhancing analytical reasoning through a flexible analytic technique

Abstract

Structured analytic techniques (SATs) have been developed to help the intelligence community reduce flaws in cognition that lead to faulty reasoning. To ascertain whether SATs provide benefits to reasoning we conducted an experiment within a web-based application, comparing three conditions: 1) unaided reasoning, 2) a prototypical order-based SAT and 3) a flexible, process-based SAT that we call TRACE. Our findings suggest that the more flexible SAT generated higher quality reasoning compared to the other conditions. Consequently, techniques and training that support flexible analytical processes rather than those that require a set sequence of steps may be more beneficial to intelligence analysis and complex reasoning.

Critical review of the Analysis of Competing Hypotheses technique: lessons for the intelligence community

“…Using various other terms, and searching through the seven entries, we identified another two entries describing experiments which appeared to satisfy the criteria. However, after closer inspection, two of the nine entries were not distinct 32 , and one of the other articles tested an SAT which, while ostensibly a version of ACH, departed from important steps in Table 1 and was therefore omitted from this review 33 . This left a total of seven distinct entries describing six experiments that were included in our review.…”

Effects of task structure and confirmation bias in alternative hypotheses evaluation

“…Many intelligence organizations train their analysts to employ SATs (for a review see Dhami et al, 2016 ), and there is particular enthusiasm for ACH, which aims to help analysts overcome confirmation bias when evaluating alternative hypotheses (UK Ministry of Defence, 2013 ; US Government, 2009 ). Recent efforts to improve analytic performance involve embedding elements of ACH, without questioning its utility, in a web-based application that affords analysts some flexibility (Stromer-Galley et al, 2021 ). Indeed, SATs such as ACH are rarely empirically evaluated and/or evidence-based.…”

“…In sum, a psychologically evidence-based approach could lead to more effective policies and practices in intelligence analysis, that could ultimately reduce the likelihood of biases and errors, while also increasing accountability processes (Dhami et al, 2015 ). Empirically evaluating SATs such as ACH is timely in light of continued recommendations for their use to solve intelligence problems (e.g., Coulthart, 2017 ; Davies & Gustafson, 2017 ; Hart, 2014 ; Lemay & Leblanc, 2018 ; Stromer-Galley et al, 2021 ). The fact that there is a proliferation of SATs such as ACH beyond the national intelligence domain to other domains involving analytic work such as in the legal and criminal justice system (e.g., Houck, 2020 ; Townsley et al, 2011 ) also means that research akin to that presented here is potentially widely applicable.…”

“…The understanding gleaned from addressing these issues can be used to reconsider the resources invested in training analysts to use existing SATs such as ACH and can contribute to the development of new, psychologically evidence-based interventions (see also Belton & Dhami, 2021 ). Such research is timely in light of continued recommendations for the use of SATs to solve intelligence problems (e.g., Coulthart, 2017 ; Davies & Gustafson, 2017 ; Hart, 2014 ; Lemay & Leblanc, 2018 ; Stromer-Galley et al, 2021 ). In addition, the proliferation of SATs beyond the national intelligence domain to law enforcement and the business intelligence sector (e.g., Houck, 2020 ; Townsley et al, 2011 ; see also Maegherman et al, 2020 ) reinforces the need to ensure that analytic policies and practices in these other domains benefit from the emerging psychological evidence base.…”