Using Noisy Self-Reports to Predict Twitter User Demographics

Abstract

Computational social science studies often contextualize content analysis within standard demographics.Since demographics are unavailable on many social media platforms (e.g. Twitter), numerous studies have inferred demographics automatically. Despite many studies presenting proof-of-concept inference of race and ethnicity, training of practical systems remains elusive since there are few annotated datasets. Existing datasets are small, inaccurate, or fail to cover the four most common racial and ethnic groups in the United States. We present a method to identify self-reports of race and ethnicity from Twitter profile descriptions. Despite the noise of automated supervision, our self-report datasets enable improvements in classification performance on gold standard self-report survey data. The result is a reproducible method for creating large-scale training resources for race and ethnicity. 1 https://bitbucket.org/mdredze/demographer 2 http://www.cs.jhu.edu/ ~mdredze/demographics-training-data/

Gender and Racial Fairness in Depression Research using Social Media

“…However, many individuals do not fit in these gender categories, some present a gender online inconsistent to their true identity (Nilizadeh et al, 2016), and they often experience depression and other mental health conditions at a higher rate (McDonald, 2018). For race/ethnicity labels, we consider the mutually-exclusive labeling conventions invoked by Brody et al (2018) and Wood-Doughty et al (2020): non-Hispanic White, non-Hispanic Black, non-Hispanic Asian and Hispanic/Latinx, as they are representative of the majority of racial and ethnic identities in the US. Our racial/ethnic categories do not capture multiracial individuals or those with a race/ethnicity outside this group.…”

“…Their classifier achieves an accuracy of 82.3% within intrinsic evaluations and shows even more promise as highconfidence thresholds are applied. To validate and further reduce noise in previously-inferred gender attributes, we train a new gender inference model on data from Burger et al (2011) using the same architecture of Wood-Doughty et al (2020). Our classifier obtains an accuracy of 83.3% amongst within-distribution data and outputs a distribution of inferred gender attributes that strongly aligns with that of the original datasets.…”

“…• Name Labels: Only considers users from CLPSYCH who could be de-anonymized; demographics annotated using name-based gender (Wood-Doughty et al, 2018) and ethnicity classifiers (Wood-Doughty et al, 2020).…”

“…Demographic Labels. Due to dataset limitations, the ethnicity and gender labels for this study were inferred using the unigram model from (Wood-Doughty et al, 2020). This model considers only the four largest race/ethnicity groups in the US, which aligns with conventions from Brody et al (2018) but ignores smaller populations and multi-racial categories.…”

“…With the permission of , we were able to collect name attributes for 622 of individuals in CLPSYCH. To obtain demographic labels from name attributes we used the demographer's neural name classifier (Wood-Doughty et al, 2018), and ethnic/race name classifier (Wood-Doughty et al, 2020). Due to the small number of individuals that we could obtain name attributes, our analysis of this technique is limited.…”