Analysis tools in the study of distributed decision-making: a meta-study of command and control research

Abstract

Our understanding of distributed decision making in professional teams and their performance comes in part from studies in which researchers gather and process information about the communications and actions of teams. In many cases, the data sets available for analysis are large, unwieldy and require methods for exploratory and dynamic management of data. In this paper, we report the results of interviewing eight researchers on their work process when conducting such analyses and their use of support tools in this process. Our aim with the study was to gain an understanding of their workflow when studying distributed decision making in teams, and specifically how automated pattern extraction tools could be of use in their work. Based on an analysis of the interviews, we elicited three issues of concern related to the use of support tools in analysis: focusing on a subset of data to study, drawing conclusions from data and understanding tool limitations. Together, these three issues point to two observations regarding tool use that are of specific relevance to the design of intelligent support tools based on pattern extraction: open-endedness and transparency.

Automated text-based analysis for decision-making research

“…Thus, using text-based classification of messages was considered an viable option for filtering the datasets used in analysis. Based on earlier work on the feasibility of using automatic pattern extraction in texts for supporting data exploration (Leifler and Eriksson, 2010b), and design criteria elicited from interviews with researchers in command and control (Leifler and Eriksson, 2010a), we designed a prototype tool to create patterns from texts. 2.…”

“…The Infomat can be considered a support tool for experts in data mining who have in-depth knowledge of both how to perform clustering, represent terms and texts in a grid-like representation and understand the significance of graphical patterns that emerge as part of manipulating the different clustering settings in the tool. One of the conclusions from our earlier interview study (Leifler and Eriksson, 2010a) was that transparency is crucial to success for advanced support tools in analysis. The Infomat tool, though capable in clustering and one of the more interactive tools available for text clustering, still required a radically different set of interactions and manual translations to the types of data normally encountered in analysis when tested on data from the scenarios used.…”

“…To facilitate the process of reducing data sources to manageable and comprehensible chunks, researchers have devised tools for visual exploration of patterns (Albinsson and Morin, 2002) to find critical incidents by using explicitly available attributes of communications to elicit patterns. From an earlier study on the work process involved in C 2 research (Leifler and Eriksson, 2010a), we had concluded that there were issues related to focusing a study and drawing conclusions from data that could probably benefit from support tools built on foundations from the data mining community.…”

“…In our design of the support tool, we built on results from an interview study which indicated that researchers spend much of their effort on narrowing research questions and looking for patterns in data (Leifler and Eriksson, 2010a) with little use from automatic tools for pattern extraction. Leifler and Eriksson (2010a) provided two main observations related to the design of intelligent support tools in analysis: the requirements for open-endedness and transparency in tool design.…”

“…Leifler and Eriksson (2010a) provided two main observations related to the design of intelligent support tools in analysis: the requirements for open-endedness and transparency in tool design. The design of the Workflow Visualizer was based on concrete interpretations of these two observations in the following way with respect to the use of text clustering:…”