Does AI Debias Recruitment? Race, Gender, and AI’s “Eradication of Difference”

Abstract

In this paper, we analyze two key claims offered by recruitment AI companies in relation to the development and deployment of AI-powered HR tools: (1) recruitment AI can objectively assess candidates by removing gender and race from their systems, and (2) this removal of gender and race will make recruitment fairer, help customers attain their DEI goals, and lay the foundations for a truly meritocratic culture to thrive within an organization. We argue that these claims are misleading for four reasons: First, attempts to “strip” gender and race from AI systems often misunderstand what gender and race are, casting them as isolatable attributes rather than broader systems of power. Second, the attempted outsourcing of “diversity work” to AI-powered hiring tools may unintentionally entrench cultures of inequality and discrimination by failing to address the systemic problems within organizations. Third, AI hiring tools’ supposedly neutral assessment of candidates’ traits belie the power relationship between the observer and the observed. Specifically, the racialized history of character analysis and its associated processes of classification and categorization play into longer histories of taxonomical sorting and reflect the current demands and desires of the job market, even when not explicitly conducted along the lines of gender and race. Fourth, recruitment AI tools help produce the “ideal candidate” that they supposedly identify through by constructing associations between words and people’s bodies. From these four conclusions outlined above, we offer three key recommendations to AI HR firms, their customers, and policy makers going forward.

Brain versus bot: Distinguishing letters of recommendation authored by humans compared with artificial intelligence

“…The potential role for AI in crafting LORs has been previously suggested; however, this is the first study that demonstrates the feasibility of this application for AI 12,13 . Human‐authored letters have documented, problematic tendencies toward bias, and AI is suggested as a potential solution to reducing bias in LOR, though contrary arguments and data exist 3,11,14–18 . Although our study only examined biased language related to gender and was not powered to examine this aspect in detail, we observed signal of gender‐biased language in both human and AI‐authored LORs.…”

“…12,13 Human-authored letters have documented, problematic tendencies toward bias, and AI is suggested as a potential solution to reducing bias in LOR, though contrary arguments and data exist. 3,11,[14][15][16][17][18] Although our study only examined biased language related to gender and was not powered to examine this aspect in detail, we observed signal of gender-biased language in both human and AI-authored LORs. Current and future work in understanding AI systems' bias will hopefully further determine whether AI can mitigate or exacerbate bias in letter writing.…”

Can AI Mitigate Bias in Writing Letters of Recommendation?

“…The other half of this process is recommendation letter reading and interpretation. Regardless of self-generated text or AI-assisted generation of text, there is a history of bias in AI-supported hiring [ 43 ]. Even human screeners are not immune to this bias, tending to carry biases when they, for example, perceive a name to be identifying a person's gender or race [ 44 , 45 ].…”

Gender equity in hiring: examining the effectiveness of a personality-based algorithm

“…Despite these findings, many researchers have sounded the alarm. For instance, Drage and Mackereth (2022) , in their review of assertions made by AI providers, suggested that endeavors to eradicate gender and race from AI frequently misinterpret these concepts as discrete characteristics rather than broader structures of power, or that that using AI as a fix for gender diversity issues, an example of technosolutionism, fails to address the inherent systemic issues within organizations. Others raise concerns that algorithms could unintentionally exacerbate existing biases within recruitment processes ( Kelly-Lyth, 2021 ).…”