Taming Human Subjects: Researchers’ Strategies for Coping with Vagaries in Social Science Experiments

Abstract

The experimental method is designed to secure the reliable attribution of causal relationships by means of controlled comparison across conditions. Doing so, however, depends upon the reduction of uncertainties and inconsistencies in the process of comparison; and this poses particularly significant challenges for the behavioral and social sciences because they work with human subjects, whose malleability and complexity often interact in unexpected ways with experimental manipulations, thus resulting in unpredictable behavior. Drawing on the Science and Technology Studies perspective and one of our author's experiences in experimental work, this paper examines how experimental social scientists manage to establish objectivity and standardization in the face of vagaries arising from working with human subjects. In identifying experimental researchers' solutions to this challenge, we draw on methodological discussions among applied social scientists as naturally occurring data, through which we show how some seemingly mundane practices play essential roles in extracting patterns out of otherwise unpredictable behaviors in the lab. Closely examining such strategies, we reveal the inherent instabilities in the experimental method when adopted in the social sciences and discuss their methodological implications. In conclusion, we make tentative suggestions for escaping the kinds of methodological impasse which we have identified.

Habeas Corpus? Cultural Keywords, Statistical Keywords, and the Role of a Corpus in their Identification

“…Of course, once the corpus has been designed, other kinds of restriction or filtering must also come early into play: 'we were selective about the articles we retrieved, and included only news articles that dealt with political or current affairs' (p. 24). Thus, 'We achieved this in part by selecting only those articles which contained specific search terms', such as Labour AND/OR Blair AND/OR Government (p. 25). What counts as an article, however, is not made clear.…”

“…If auxiliary assumptions are implicit and unrecognised in the design of a research method, then these unstated assumptions make it difficult for subsequent researchers to reproduce the method in the way the original researchers applied it. (See Ting and Montgomery 25 ). In the fields of experimental psychology or social psychology, for instance, failures of replicability (now a source of much concern) can be traced in part to 'insufficient specification of the conditions necessary or sufficient to obtain the results'.…”