More on why Lakisha and Jamal didn’t get interviews: Extending previous findings through a reproducibility study

Abstract

In 2004, Bertrand and Mullainathan published an innovative piece of research that involved sending nearly 5,000 fictitious resumes to employers. Their paper is commonly cited for finding that applicants with White-sounding names benefitted more from experience on their resumes and received 50 percent more invitations to interview than other applicants. The current research, however, demonstrates that while Bertrand and Mullainathan made a critically important contribution to the literature on employment discrimination, there is still much more that we can learn from their data. Through a reanalysis of Bertrand and Mullainathan's data, we find that discriminatory effects were stronger in conditions in which a job posting's experience requirements were ambiguous, applicants with first names of Arabic origin experienced higher levels of discrimination than applicants with other non-White-sounding names, and that the discriminatory effects of having an African-American-sounding name could not be empirically differentiated from the discriminatory effects associated with a name's frequency within the overall population. Our findings contribute to the literature on human information processing and offer important practical contributions regarding how employers can potentially reduce discrimination in selection processes. Additionally, we offer important suggestions for the development of racial manipulations in future experimental research.

Designing, executing, and publishing replication research: Best practices for successfully taking replication ideas from conceptualization to publication

“…When space permits, a table that lists hypotheses, whether each hypothesis was supported, and the corresponding statistical output (e.g., path coefficients, effect sizes, p -values) for each study allows the reader to better understand how the replication and the target study fit together within the literature (see Figure 3 for an example). When there are a large number of different analyses performed (e.g., Obenauer, 2023), you may format the table such that each analysis or study is associated with only one column (see Figure 4 for an example). Sometimes, a table that simply shows what hypotheses were supported in each study can be more practical (e.g., Obenauer & Kalsher, 2023).…”

“…For example, when conducting literal replication, the researcher can simply restate the hypotheses from the target study that they will be retesting (e.g., Bermiss et al, 2023; Hamdani et al, 2023; Obenauer & Kalsher, 2023; Obenauer & Rezaei, 2023). When introducing new variables, it is helpful if the researcher motivates the introduction of these variables and proposes new hypotheses to test that will generate an understanding of the impact of these variables (e.g., Hammond et al, 2023; Obenauer, 2023; Obenauer & Rezaei, 2023). Choosing not to articulate hypotheses and utilize a research question is also possible (e.g., Chang et al, 2016; Hopp & Pruschak, 2023; Kalsher et al, 2019), though this method can make defining successful replication more difficult.…”

“…For example, in Obenauer (2023), which is a constructive reproducibility study, I sought to understand how factors such as a job applicant's name having Arabic origins or the name's overall frequency within the population influenced the findings of Bertrand and Mullainathan's (2004) seminal research on the relationship between the job applicant's race and their likelihood of receiving an invitation to interview. Before testing these relationships, however, I had to demonstrate that I could reproduce the results of the original study through literal reproduction using the same data.…”

“…Again, I recommend focusing on establishing the importance of the target study along with how the current replication will contribute to the literature, rather than focusing on perceived problems with the target. For example, in Obenauer (2023, p. 116) I said: “[Target study] serves as an appropriate target for reproduction because of its high impact and relevance to current societal issues. In addition to having more than 6,600 citations on Google Scholar as of February 2023, this paper has influenced the development of academic theory regarding discrimination in selection decisions (e.g., Charles & Guryan 2008; Umphress, Simmons, Boswell, & Triana 2008), and is frequently referenced by the mainstream media (e.g., Fussell 2016; Parkinson & Smith-Walters 2015).…”

“…Rather it serves as a starting point for members of a field that recognizes the need for replication, but also lacks resources to guide researchers in replication. I will provide this guidance by drawing on my own experience publishing replication research (Kalsher et al, 2019; Obenauer, 2023; Obenauer & Kalsher, 2023; Obenauer & Rezaei, 2023) as well as patterns that I have observed in the literature.…”