C3Fire in command and control research

2011

Cogn Tech Work

Our understanding of distributed decision making in professional teams and their performance comes in part from studies in which researchers gather and process information about the communications and actions of teams. In many cases, the data sets available for analysis are large, unwieldy and require methods for exploratory and dynamic management of data. In this paper, we report the results of interviewing eight researchers on their work process when conducting such analyses and their use of support tools in this process. Our aim with the study was to gain an understanding of their workflow when studying distributed decision making in teams, and specifically how automated pattern extraction tools could be of use in their work. Based on an analysis of the interviews, we elicited three issues of concern related to the use of support tools in analysis: focusing on a subset of data to study, drawing conclusions from data and understanding tool limitations. Together, these three issues point to two observations regarding tool use that are of specific relevance to the design of intelligent support tools based on pattern extraction: open-endedness and transparency.

Section: Researchmentioning

confidence: 99%

Analysis tools in the study of distributed decision-making: a meta-study of command and control research

2011

Cogn Tech Work

“…During the scenario, commanders operated at three levels of command, in two administrative areas (approximately county-sized areas) and played a role-playing simulation exercise (Trnka and Jenvald 2006) in which there was initially a forest fire but subsequently also an evacuation from a zoo as well as a search and rescue operation. Participants communicated with one another through a text-based messaging system designed for use in micro-world simulations with the C 3 Fire simulation environment (Johansson et al 2003), although it shared the basic features of an e-mail messaging system without the use of subject lines or other auxiliary e-mail headers. The scenario was played over the course of one day.…”

Section: Alfa -05mentioning

confidence: 99%

“…First, understanding how command teams and team members perform their tasks means constructing a general workflow model such as DOODA from command and control scenarios. Second, establishing direct performance measures is synonymous to relating the workflow model to the estimated outcome of scenarios as defined by indirect measurements of scenario outcome (for example, performance scores in computer simulations (Johansson et al 2003) or evaluations by human experts of team performance in role-playing exercises (Jensen 2009)). Third, improving performance is equal to, in each particular scenario, using those performance measures to relate staff actions to the proposed workflow.…”

Section: Introductionmentioning

confidence: 99%

Message classification as a basis for studying command and control communications—an evaluation of machine learning approaches

2011

J Intell Inf Syst

Specifically, we report the results from evaluating machine leaning with respect to two metrics of classification performance: (1) the precision of finding a known transition between two activities in a work process, and (2) the precision of classifying messages similarly to human researchers that search for critical episodes in a workflow.The results indicate that classification based on text only provides higher precision results with respect to both metrics when compared to other machine learning approaches, and that the precision of classifying messages using text-based classification in already classified datasets was approximately 50%. We present the implications that these results have for the design of support systems based on machine learning, and outline how to practically use text classification for analyzing team communications by demonstrating a specific prototype support tool for workflow analysis.

“…In the research projects we have studied, two such methods have been in use: roleplaying simulations (Rubel, 2001) and micro-world simulations (e.g., (Waern and Cañas, 2003;Johansson et al, 2003)), both of which imply certain characteristics of the data sets generates and require specific methods of analysis.…”

Section: Conclusion Drawing/verifyingmentioning

confidence: 99%

Automated text-based analysis for decision-making research

2011

Cogn Tech Work

We present results from a study on constructing and evaluating a support tool for the extraction of patterns in distributed decision making processes, based on design criteria elicited from a study on the work process involved in studying such decision making. Specifically, we devised and evaluated an analysis tool for C 2 researchers who study simulated decision-making scenarios for command teams. The analysis tool used text clustering as an underlying pattern extraction technique, and was evaluated together with C 2 researchers in a workshop to establish whether the design criteria were valid and the approach taken with the analysis tool was sound.Design criteria elicited from an earlier study with researchers (open-endedness and transparency) were highly consistent with the results from the workshop. Specifically, evaluation results indicate that successful deployment of advanced analysis tools require that tools can treat multiple data sources and offer rich opportunities for manipulation and interaction (open-endedness) and careful design of visual presentations and explanations of the techniques used (transparency). Finally, the results point to the high relevance and promise of using text clustering as a support for analysis of C 2 data.