Scoping the Emerging Field of Quantitative Ethnography: Opportunities, Challenges and Future Directions

Abstract

Quantitative Ethnography (QE) is an emerging methodological approach that combines ethnographic and statistical tools to analyze both Big Data and smaller data to study human behavior and interactions. This paper presents a methodological scoping review of 60 studies employing QE approaches with an intention to characterize and establish where the boundaries of QE might and should be in order to establish the identity of the field. The key finding is that QE researchers have enough commonality in their approach to the analysis of human behavior with a strong focus on grounded analysis, the validity of codes and consistency between quantitative models and qualitative analysis. Nonetheless, in order to reach a larger audience, the QE community should attend to a number of conceptual and methodological issues (e.g. interpretability). We believe that the strength of work from individual researchers reported in this review and initiatives such as the recently established International Society for Quantitative Ethnography (ISQE) can present a powerful force to shape the identity of the QE community.

Methodology in the Mirror: A Living, Systematic Review of Works in Quantitative Ethnography

“…Of the 66 studies included where IRR would have been applicable, about half calculated it, usually via Cohen's Kappa, and about half did not report whether this was performed. Kaliisa et al coded for various types of code validation, but arrived to similar conclusions (n=23/60), adding that many authors did not report the exact statistical procedure applied to compute IRR or did not disclose other forms of validation [13].…”

“…QE methods are applicable to a range of discourse data, but the majority of studies in our sample involved text data, most commonly from observations, archives, chatroom discussions, or interviews. A scoping review by Kaliisa et al [13] examining the boundaries, opportunities, and challenges of QE analyzed 60 papers, similarly noting that most QE studies worked with text-based raw data and that non-text data (e.g., videos) were always converted to text. This points to a facet of QE methodology where, albeit raw discourse data can be audio or visual in nature, in order to perform analyses, it must be transformed, most commonly transcribed, which involves a series of crucial decisions.…”

“…If QE asserts being a unified methodology, with the ability to make both qualitative and quantitative claims, should there be more emphasis on sampling in QE studies? Kaliisa et al note that 44 out of 60 reviewed QE studies had a sample size of less than 100, and argue that this "small sample size" can lead to challenges in drawing conclusions and integrating statistical approaches, as well as affect the application of QE at scale [13]. One could also argue that this is not an issue of sample size, but of sampling: if the sample was not chosen at random from the target population, then it is the sampling procedure that prevents generalization to the wider population, not the size of the sample.…”

“…Formulating justifications for these decisions and reporting on them (or creating supplementary materials) could be beneficial not only for the authors but the QE community as well. [13]. This may be because if a methodology is in such close dialectic with a tool, it is difficult to separate the two.…”

A Systematic Literature Review of Empirical Research on Epistemic Network Analysis in Education

“…The network visualization of ENA models the cooccurrence of, for example, codes in discourse, activities in log files, or elements of interaction in a chat [3]. Whereas the method has been conceptualized in education research, it has been used in a wide variety of research questions and applications [4]. Recently, ENA became part of a growing new community of quantitative ethnography that extends to different types of human behavior and applications [4].…”