Comparing Naval Decision Support Technologies Using Decision Models, Process Tracing and Error Analysis

Abstract

Providing decision support to operators in command and control contexts requires careful assessment of its impacts on task performance. Here we describe a human-in-the-loop experiment using a naval air defense testbed to compare three conditions: 1) a baseline interface (DSSBASE; 2) one that displays the temporal proximity of radar aircraft (DSSTEMP); and 3) one which adds a change history panel (DSSCHEX). Threatevaluation accuracy and cognitive models of participants’ judgments did not significantly differ across conditions. Eye-tracking data showed that DSSTEMP lead to a reduced verification of track attributes. Confusion matrices also differed across conditions: DSSTEMP lead to more errors that are “two-categoriesaway” when erroneously classifying hostile aircraft as non-hostile or uncertain. We conclude that in addition to looking at standard metrics of task accuracy and response times, critical insights can be obtained by assessing how different designs for decision support may alter strategies and cognitive processes.

Combining Process Tracing and Policy Capturing Techniques for Judgment Analysis in an Anti-Submarine Warfare Simulation

“…Among them is the information board method, which involves monitoring the type of information that an individual selects before making a decision as well as the order in which it is selected (Cook & Swain, 1993). Given their complementarity, previous work highlighted the potential advantages of using PT and policy capturing techniques together to better extract decision processes (Billings & Marcus, 1983;Lafond, Champagne, et al, 2009;Lafond et al, 2017). For example, in Lafond, Lacouture, and Cohen (2009), process tracing data allowed for specifying the structure of individual decision process models, which better predicted human decisions, errors and response times.…”