Methods to Establish Race or Ethnicity of Twitter Users: Scoping Review

Abstract

Background A growing amount of health research uses social media data. Those critical of social media research often cite that it may be unrepresentative of the population; however, the suitability of social media data in digital epidemiology is more nuanced. Identifying the demographics of social media users can help establish representativeness. Objective This study aims to identify the different approaches or combination of approaches to extract race or ethnicity from social media and report on the challenges of using these methods. Methods We present a scoping review to identify methods used to extract the race or ethnicity of Twitter users from Twitter data sets. We searched 17 electronic databases from the date of inception to May 15, 2021, and carried out reference checking and hand searching to identify relevant studies. Sifting of each record was performed independently by at least two researchers, with any disagreement discussed. Studies were required to extract the race or ethnicity of Twitter users using either manual or computational methods or a combination of both. Results Of the 1249 records sifted, we identified 67 (5.36%) that met our inclusion criteria. Most studies (51/67, 76%) have focused on US-based users and English language tweets (52/67, 78%). A range of data was used, including Twitter profile metadata, such as names, pictures, information from bios (including self-declarations), or location or content of the tweets. A range of methodologies was used, including manual inference, linkage to census data, commercial software, language or dialect recognition, or machine learning or natural language processing. However, not all studies have evaluated these methods. Those that evaluated these methods found accuracy to vary from 45% to 93% with significantly lower accuracy in identifying categories of people of color. The inference of race or ethnicity raises important ethical questions, which can be exacerbated by the data and methods used. The comparative accuracies of the different methods are also largely unknown. Conclusions There is no standard accepted approach or current guidelines for extracting or inferring the race or ethnicity of Twitter users. Social media researchers must carefully interpret race or ethnicity and not overpromise what can be achieved, as even manual screening is a subjective, imperfect method. Future research should establish the accuracy of methods to inform evidence-based best practice guidelines for social media researchers and be guided by concerns of equity and social justice.

A case against social networks as the only means for recruitment in scientific studies with humans

“…Namely, the likelihood of segregation in networks, an effect known as filter bubble [32,34], and the ranking signals that govern how message contents become available to users in a network [3,21,29]. Other issues have been identified derived from online recruitment, especifically participant retention [28], age bias [17] and racial and economical bias [19,31,50]. Some studies have shown that community-based methods, including personal references, local advertisements and pamphlets, indeed recruit more racially diverse samples than other methods [41], and may ensure better chances of participant retention [2,7,44,46,49].…”

“…In this sense, differences in age have be found between participants recruited via social networks * E-mail: alvaropastor@uoc.edu and other methods [17]. As well as a risk for racial and economical bias has been reported when using social networks data [19,31,50]. Moreover, the literature from a diversity of disciplines, including machine learning and data mining [6,23,36], and marketing [20,27,48], sheds light on the various management layers that aim to optimise links between users and the flow of messages.…”

Social media and COVID‐19 vaccination hesitancy during pregnancy: a mixed methods analysis

“…However, we were able to run automated detection software to determine the geolocation and age 33,34 of the Twitter users. In addition, two researchers (KO and SG) independently annotated basic race categories (black, Asian and white) 35 of Twitter users using their photos and self‐declarations in their bio or recent timelines.…”

“We Are Bulletproof”: The Transcultural Power of Fandom in #StopAsianHate

“…One additional limitation to this work is the lack of disaggregation by race or gender, especially considering that a majority of ARMY are women (Grover et al 2022). While we wish to examine these topics under a lens that incorporates these demographic characteristics, such work is difficult without making generalized inferences, and there is not a uniformly accepted method for ascertaining the race or ethnicity of social media users (Golder et al 2022;Jung et al 2018). For example, many papers in computer science that examine Internet data use facial recognition or artificial intelligence (AI) software in order to derive the gender and race of respondents from Twitter profiles.…”

“…While we wish to examine these topics under a lens that incorporates these demographic characteristics, such work is difficult without making generalized inferences, and there is not a uniformly accepted method for ascertaining the race or ethnicity of social media users (Golder et al. 2022; Jung et al. 2018).…”