Reading the Tea Leaves: Ethnographic Prediction as Evidence

Abstract

Those who work in research know that we live in a world that is strongly influenced by what Tricia Wang has called the quantification bias. More so than other forms of information, numbers have incredible formative power. In our culture, numbers are seen as trustworthy representations of reality that are strongly associated with objectivity and untainted by human bias and shortcomings. Recently, data science, big data, algorithms, and machine learning have fueled a new wave of the quantification bias. One of the central fascinations of this wave has been the promise that humans now have the power of prediction at their fingertips. In this paper, I reflect on what it means to make predictions and explore the differences in how predictions are accomplished via quantitative modeling and ethnographic observation. While this is not the first time that ethnographic work has been put in conversation and in contrast with quantified practices, most theorists have framed the role of ethnography as providing context to that quantified work. Here, I argue that ethnographers produce predictions in their own right. I begin by discussing what it means to predict something, focusing on its function. This is followed by a discussion of the ways in which predictions are constructed through both machine learning and ethnographic work. In the course of this discussion I show the commonalities that exist between ethnographic work and machine learning, and I outline methodologies that claim that ethnographic work can make generalizable and accurate statements about the world, including predictive claims. I also point to some of the challenges in using machine learning as a means of producing predictions. This discussion is not meant to discredit these practices, but to demystify the process as a means of loosening quantification's authority, contextualizing its best applications, and putting the two approaches to knowledge production on even footing. Finally, I discuss circumstances in which qualitatively produced predictions may be most valuable, such as when dealing with emerging phenomena and unstable contexts.

Building Resilient Futures in the Virtual Everyday: Virtual Worlds and the Social Resilience of Teens During COVID‐19

“…We believe that following such an approach helps us to understand how new technologies are adapted to existing practices of sociality and how they shape the cultural categories of socializing of today and tomorrow (Smits 2006, 501). By adopting an ethnographic method in a time of transition from a pandemic to a postpandemic reality we also follow an ethnofuturist approach that we think is necessary to identify enduring behaviors that point to potential futures (English-Lueck, Ladner and Sherman 2021; Maiers 2018).…”

“…We also recruited teens across a spectrum from 13-17 years old, which enabled us to compare past and present behaviors between younger and older teens to help distinguish practices that might dissipate. By delivering a rich picture of teenagers' present-day technology practices we hope our insights provide a lot more than context to future predictions based on big data (Maiers 2018) and help to make meaningful predictions about how everyday life with, in and through virtual worlds will look, sound, and feel.…”

Ethnographic Agency in a Data Driven World

“…In my own experiences, however, most of my colleagues-other user researchers included-did not understand the nature or value of qualitative data. They did not understand which business questions would best benefit from ethnographic inquiries, or that qualitative insights were never meant to be "representative" (Maiers 2018). As a result, business decisions were still largely driven by quantitative insights from surveys and by behavioral shifts seen through the lens of log data.…”

“…Because my ethnographic analysis became data-driven, my stakeholders perceived it to be more rigorous and high quality. The close relationship between quantitative and qualitative research helped to circumvent the all-too-common criticism that qualitative research lacked statistical validity or situational generalizability (Maiers 2018), Here, ethnography was not just a way to give texture to quantitative data. Instead, quantitative methods emerged as a way to give new value and life to ethnography and qualitative data itself, by leveraging "big data" to open up opportunities to explain cultural differences (Curran 2013).…”

“…This has left ethnographers, in both industry and academic settings, with few playbooks for engaging critically with quantitative data, or for doing "participant observation" on the practices that shape how quantitative data are made and used. This paper takes as its problematic the author's three years of experience working in a large, data-intensive organization, where ethnographers often struggle to gain traction with their work because of an inherent bias towards quantification and numbers (Maiers 2018). By providing an autoethnography of my experiences learning quantitative methods (Jones, Adams, and Ellis 2016), I argue that quantitative literacy-a deep familiarity with data, as well as knowledge of how to use quantitative skills to shape an organization-can give ethnographers expanded agency 2 to do research in and impact data-intensive businesses.…”