Using Machine Learning to Support Qualitative Coding in Social Science

Abstract

Machine learning (ML) has become increasingly influential to human society, yet the primary advancements and applications of ML are driven by research in only a few computational disciplines. Even applications that affect or analyze human behaviors and social structures are often developed with limited input from experts outside of computational fields. Social scientists—experts trained to examine and explain the complexity of human behavior and interactions in the world—have considerable expertise to contribute to the development of ML applications for human-generated data, and their analytic practices could benefit from more human-centered ML methods. Although a few researchers have highlighted some gaps between ML and social sciences [51, 57, 70], most discussions only focus on quantitative methods. Yet many social science disciplines rely heavily on qualitative methods to distill patterns that are challenging to discover through quantitative data. One common analysis method for qualitative data is qualitative coding . In this article, we highlight three challenges of applying ML to qualitative coding. Additionally, we utilize our experience of designing a visual analytics tool for collaborative qualitative coding to demonstrate the potential in using ML to support qualitative coding by shifting the focus to identifying ambiguity. We illustrate dimensions of ambiguity and discuss the relationship between disagreement and ambiguity. Finally, we propose three research directions to ground ML applications for social science as part of the progression toward human-centered machine learning.

Automatic Detection of Everyday Social Behaviours and Environments from Verbatim Transcripts of Daily Conversations

“…More precisely, we took every 3rd sample from the test dataset and added it to the train dataset. We then performed 3-fold cross validation 5 . The results can be seen in Fig.…”

“…The process is even further complicated by the need of producing annotation with high interrater reliability, which in itself includes training the coders for a given problem [32]. As the size of the data typically collected in studies increases [5], it becomes difficult, even impossible, to code the data manually.…”

“…Although the use of automated methods for codes analysis is widely spread in social sciences [5], the automated coding is an emerging research field. What is more, it is usually conducted by computer scientists without much involvement of the domain experts in the development of the codes [5].…”

Towards an automatic coding of observational studies: Coding neurofeedback therapies of children with autism

“…In the case of an ML approach, Abbas et al (2018) demonstrated a significant accuracy improvement over standard screening tools in a clinical study using a sample of 162 children. Despite the previous work, low precision has been considered a limitation with such approaches Chen et al (2018), therefore researchers still face challenges applying ML in this domain. Bone et al (2015) argue that clinical domain experts and computational researchers must cooperate in order to maximize the potential use of ML for behavioural science.…”

“…There is an increasing amount of work using cameras as the main sensor placed at a certain distance from the child to automatically recognise their behaviour with computer vision (CV) methods (Dawson et al, 2018; Hashemi et al, 2014, 2015; Manner, Jiang, Zhao, Gini, & Elison, 2017). Additionally, with the advances in ML, there has also been an increasing amount of work studying its application for automatic coding (Abbas, Garberson, Glover, & Wall, 2018; Bone et al, 2015; Chen, Drouhard, Kocielnik, Suh, & Aragon, 2018). There are issues that have been identified when using ML techniques, which are related to the lack of shared knowledge between problem‐domain and experts.…”