Learning from failures of protocol in cross-cultural research

Abstract

The many tools that social and behavioral scientists use to gather data from their fellow humans have, in most cases, been honed on a rarefied subset of humanity: highly educated participants with unique capacities, experiences, motivations, and social expectations. Through this honing process, researchers have developed protocols that extract information from these participants with great efficiency. However, as researchers reach out to broader populations, it is unclear whether these highly refined protocols are robust to cultural differences in skills, motivations, and expected modes of social interaction. In this paper, we illustrate the kinds of mismatches that can arise when using these highly refined protocols with nontypical populations by describing our experience translating an apparently simple social discounting protocol to work in rural Bangladesh. Multiple rounds of piloting and revision revealed a number of tacit assumptions about how participants should perceive, understand, and respond to key elements of the protocol. These included facility with numbers, letters, abstract number lines, and 2D geometric shapes, and the treatment of decisions as a series of isolated events. Through onthe-ground observation and a collaborative refinement process, we developed a protocol that worked both in Bangladesh and among US college students. More systematic study of the process of adapting common protocols to new contexts will provide valuable information about the range of skills, motivations, and modes of interaction that participants bring to studies as we develop a more diverse and inclusive social and behavioral science. generalizability | diversity | cross-cultural | social discounting | Bangladesh I n 1932, the psychologist Rensis Likert (1) published his dissertation on a novel method for measuring attitudes. After giving university students printed statements about race relations, Likert asked them to check one of five options (strongly approve, approve, undecided, disapprove, and strongly disapprove) indicating how much they endorsed each of these statements. Likert then assigned numbers to these levels of approval and took an average across all statements. The simplicity of both the response format and construction of the scale soon spurred researchers to adopt elements of the technique to assess not only attitudes (Likert's original interest) but also subjective judgments along many dimensions, including likelihood, desirability, difficulty, and happiness (2). Today, after decades of testing and refinement on generations of participants, Likert's simple format has become a reliable mainstay of social and behavioral research.Given its ubiquity in the social and behavioral sciences, one might guess that a five-or seven-item Likert format is a natural way of asking humans about their subjective judgments. However, in the rare cases when researchers have described their experience using Likert items outside of formally educated populations, they have been met with mixed success (3,4). It turns out tha...

Generosity and livelihoods: Dictator game evidence on the multidimensional nature of sharing among the Kenyan Maasai

“…With regard to social distance, we could replicate in a non‐WEIRD population (the Kenyan Maasai) previous social discounting studies with WEIRD samples (Jones & Rachlin, ; Strombach et al, , ; Goeree et al ; Margittai et al, , ) that generosity towards others declines across social distance. Our social discounting results are consistent with those obtained from other non‐WEIRD populations, including Indian (Hackmann, Danvers & Hruschka, ), Singaporean (Pornpattananangkul et al, ), and Banghadeshi participants (Hruschka et al, ), but they extended previous results by the observation that social discounting was not uniform across all goods and commodities. Some resources like grass and money are more readily shared than other resources such as milk and cows between people considered socially close .…”

“…Singaporean (Pornpattananangkul et al, 2017), and Banghadeshi participants (Hruschka et al, 2018), but they extended previous results by the observation that social discounting was not uniform across all goods and commodities. Some resources like grass and money are more readily shared than other resources such as milk and cows between people considered socially close.…”

Social information use in adolescents: the impact of adults, peers and household composition

“…Social information use in adolescents the paradigm used in our study might form a sound basis for systematic comparisons of the development of social learning across societies (in the spirit of, e.g., [46]), disentangling the effects of age and socio-cultural environment. For useful discussions of some of the challenges of using cognitive tasks across cultures, see [76][77][78].…”

Biological Anthropology in 2018: Grounded in Theory, Questioning Contexts, Embracing Innovation

“…Notably, both Hruschka et al. () and Winking et al. () cite results from their own external cross‐checks in concluding that the differences they observed in their study populations relative to standard findings likely did not derive from differences in protocol delivery.…”

“…Hruschka et al. () review the many problems in adapting research tools, such as Likert scales, that were developed within formally educated populations for use in more culturally and economically varied settings—further noting that there has been little systematic examination of what specific tools generate meaningful or unusable responses in different contexts. Drawing from their work adapting a well‐established experimental protocol for rural Bangladeshi study communities, Hruschka et al.…”