Message classification as a basis for studying command and control communications—an evaluation of machine learning approaches

Abstract

Specifically, we report the results from evaluating machine leaning with respect to two metrics of classification performance: (1) the precision of finding a known transition between two activities in a work process, and (2) the precision of classifying messages similarly to human researchers that search for critical episodes in a workflow.The results indicate that classification based on text only provides higher precision results with respect to both metrics when compared to other machine learning approaches, and that the precision of classifying messages using text-based classification in already classified datasets was approximately 50%. We present the implications that these results have for the design of support systems based on machine learning, and outline how to practically use text classification for analyzing team communications by demonstrating a specific prototype support tool for workflow analysis.

Automated text-based analysis for decision-making research

“…The annotation schema creates a set of distinct objects of study in the form of a set of episodes, which in turn direct further analysis by making video logs or observer reports at specific points in time relevant to study. The four stages of Rosell and Velupillai could similarly be described as re-formulations of the stages in qualitative data analysis introduced by Miles. Due to the similarities between the paradigm of data mining research and data analysis in decision making research, we had previously conducted a study on using data mining methods in text analysis (Leifler and Eriksson, 2010b). In our evaluations, we found that of all the metadata found in communications between members of a group of decision makers, the message texts were the most significant factors when attempting to emulate human classifications by machine classification.…”

“…Thus, using text-based classification of messages was considered an viable option for filtering the datasets used in analysis. Based on earlier work on the feasibility of using automatic pattern extraction in texts for supporting data exploration (Leifler and Eriksson, 2010b), and design criteria elicited from interviews with researchers in command and control (Leifler and Eriksson, 2010a), we designed a prototype tool to create patterns from texts. 2.…”

“…We had previously found text clustering in particular to be useful as a pattern extraction technique compared to other techniques for extraction information from communications and observer reports (Leifler and Eriksson, 2010b). Text clustering can be performed to relate texts to one another based on distance metrics and when suitably used in a framework for clustering texts based on them, they can be used to guide a manual search for patterns between texts and terms in texts, as has been demonstrated in the Infomat information exploration tool by Rosell and Velupillai (2008).…”

Analysis tools in the study of distributed decision-making: a meta-study of command and control research

“…Due to the similarities between these two paradigms, we had previously evaluated the technical soundness of using data mining methods in text analysis (Leifler and Eriksson, 2010) to support the exploration of patterns. We argue that, as the work process described by the ESDA model coincides well with the scatter/gather paradigm of how data mining tools are to be used, tools for data mining could probably fit the tasks in ESDA well.…”

“…In our previous study on classification of messages and observer reports from decision-making scenarios, we found text clustering in particular to be useful as a pattern extraction technique (Leifler and Eriksson, 2010). Text clustering can be used to relate texts to one another based on distance metrics.…”