Judgment Analysis in a Dynamic Multitask Environment: Capturing Nonlinear Policies Using Decision Trees

Abstract

Policy capturing is a judgment analysis method that typically uses linear statistical modeling to estimate expert judgments. A variant to this technique is to capture decision policies using data-mining algorithms designed to handle nonlinear decision rules, missing attributes, and noisy data. In the current study, we tested the effectiveness of a decision-tree induction algorithm and an instance-based classification method for policy capturing in comparison to the standard linear approach. Decision trees are relevant in naturalistic decision-making contexts since they can be used to represent "fast-and-frugal" judgment heuristics, which are well suited to describe human cognition under time pressure. We examined human classification behavior using a simulated naval air defense task in order to empirically compare the C4.5 decision-tree algorithm, the k-nearest neighbors algorithm, and linear regression on their ability to capture individual decision policies. Results show that C4.5 outperformed the other methods in terms of goodness of fit and cross-validation accuracy. Decisiontree models of individuals' judgment policies actually classified contacts more accurately than their human counterparts, resulting in a threefold reduction in error rates. We conclude that a decision-tree induction algorithm can yield useful models for training and decision support applications, and we discuss the application of judgmental bootstrapping in real time in dynamic environments.

See no evil: Cognitive challenges of security surveillance and monitoring.

“…Tracking and eliciting such tacit knowledge-and so-called intuitive decisionmaking-have proven quite difficult (see Sauter, 1999) despite the availability of a range of knowledge elicitation techniques (Hoffman, 2008). One potential avenue is the use of policy capturing and process tracing techniques in order to extract the decision path that leads to investigating an event further (see Kim, Chung, & Paradice, 1997;Lafond et al, 2009;Lafond, Vallières, Vachon, & Tremblay, 2017).…”

“…As noted previously, experienced CCTV operators often report having a “sixth sense” for when an incident will occur; it would therefore be useful for training purposes to somehow capture the features that operators most frequently use in order to detect criminal activity. While it may be difficult for operators to verbalize the precise cues that they are picking up on (see, e.g., Cooke, 1999), as mentioned before, it might be possible to employ a policy capturing approach—a judgement analysis method using statistical models or machine learning algorithms to estimate expert judgements (e.g., Lafond et al, 2017)—to help begin to understand a little more about the subjective decision criteria that are used for identifying suspicious activity in such cases. Eye-movement data could enable tracing the areas most recently fixated on leading up to a decision, as well as those areas fixated on most frequently and for the longest (see Rehder & Hoffman, 2005).…”

Improving Ambulance Dispatching with Machine Learning and Simulation

“…They employ machine learning to match expert decisions with the objective of decreasing dependency on human resources. Lafond et al [5] compare three machine learning techniques in capturing human classification behavior using a simulated naval air defense task. However, capturing expert decisions with the objective to support future decisions implicitly assumes that the captured expert knowledge is optimal, or at least neglects the fact that insight into current practices provides a good opportunity for the identification and evaluation of improvement of the decision making process.…”

Comparing Naval Decision Support Technologies Using Decision Models, Process Tracing and Error Analysis

“…Decision trees are formally equivalent to logical proposition, yet also represent a sequential feature-verification process indicating how the rule is applied. Decision trees are particularly relevant in naturalistic decision-making contexts since they can be used to represent "fast-and-frugal" (as opposed to exhaustive) judgment heuristics, which are well suited to describe human cognition under time pressure (Lafond, Vallières, Vachon, & Tremblay, 2016).…”