Avoiding bias when inferring race using name-based approaches

Abstract

Racial disparity in academia is a widely acknowledged problem. The quantitative understanding of racial-based systemic inequalities is an important step towards a more equitable research system. However, because of the lack of robust information on authors’ race, few large-scale analyses have been performed on this topic. Algorithmic approaches offer one solution, using known information about authors, such as their names, to infer their perceived race. As with any other algorithm, the process of racial inference can generate biases if it is not carefully considered. The goal of this article is to assess the extent to which algorithmic bias is introduced using different approaches for name-based racial inference. We use information from the U.S. Census and mortgage applications to infer the race of U.S. affiliated authors in the Web of Science. We estimate the effects of using given and family names, thresholds or continuous distributions, and imputation. Our results demonstrate that the validity of name-based inference varies by race/ethnicity and that threshold approaches underestimate Black authors and overestimate White authors. We conclude with recommendations to avoid potential biases. This article lays the foundation for more systematic and less-biased investigations into racial disparities in science.

Name-Based Demographic Inference and the Unequal Distribution of Misrecognition

“…The underrepresentation of Black people in most data sets means their race will be misrecognized more often than their white peers. 24 Moreover, within the US Black population, migration, social trends and movements, class, and other factors shape who goes by distinctively Black names, and thus who is ascribed Black identity by other people and algorithms. 16 Among the Black social scientists in our sample, those whose parent(s) have PhDs were correctly recognized as Black more often than those whose parent(s) did not attend college.…”

“…Even attempts to correct for these inequalities in error rates can be thrown off by them. For example, Kozlowski et al's 24 approach to compensating for the high rate at which Black people are racially mislabeled assumes they are all mislabeled at the same rate. If their corrected data was used in an analysis of parental education or class, however, the uneven rates of racial misclassification we show for parental education would still confound the analysis.…”

“…12 Because maximum likelihood approaches to racial/ethnic categorization do not perform equally well across categories, some approaches sacrifice overall accuracy in exchange for more equal error rates across groups by changing the classification thresholds. 24 Researchers interested in aggregate estimates rather than individual labels can improve performance by using the predicted probabilities (e.g., .77 woman) rather than discrete classifications (e.g., 1 = woman).…”

Intersectional Inequalities in Science

“…In other words, we do not assign authors to a unique racial category. In previous work (58), we have shown that, given the overlap of Black and White family names (59,60), the use of a threshold-filtering those names with a probability for a single group above a threshold and assigning all authors with that name to that single category-underestimates the proportion of Black authors. This distinction is critical: We do not aim to identify each author's self-perceived racial category but to build aggregates of racial group disparities.…”

Gender gap among highly cited researchers, 2014–2021

“…When necessary, we relied on images. A limitation of this method and the study is that we used only a binary gender classification (men-women) and did not consider other genders or groups (Kozlowski et al, 2022).…”