Investigating African-American Vernacular English in Transformer-Based Text Generation

Abstract

The growth of social media has encouraged the written use of African American Vernacular English (AAVE), which has traditionally been used only in oral contexts. However, NLP models have historically been developed using dominant English varieties, such as Standard American English (SAE), due to text corpora availability. We investigate the performance of GPT-2 on AAVE text by creating a dataset of intent-equivalent parallel AAVE/SAE tweet pairs, thereby isolating syntactic structure and AAVE-or SAE-specific language for each pair. We evaluate each sample and its GPT-2 generated text with pretrained sentiment classifiers and find that while AAVE text results in more classifications of negative sentiment than SAE, the use of GPT-2 generally increases occurrences of positive sentiment for both. Additionally, we conduct human evaluation of AAVE and SAE text generated with GPT-2 to compare contextual rigor and overall quality.

Societal Biases in Language Generation: Progress and Challenges

“…We infer that the less constrained, opendomain nature of continuation generation tasks makes it more preferable to evaluate mitigation through more flexible comparisons rather than absolute scores. For autocomplete generation, Sheng et al (2019 and Groenwold et al (2020) compare regard or sentiment scores across demographics, Shwartz et al (2020) compare names across various intermediate metrics, Vig et al (2020) measure proportional differences between the amount of bias under a gendered versus ambiguous reading, and Yeo and Chen (2020) compare occupations generated for different genders. Bias studies in dialogue generation use relative scores by comparing sentiment and offensive language discrepancies (Henderson et al, 2018;Liu et al, 2020a,b) and the percentage of gendered words (Dinan et al, 2020a).…”

“…Because of the difficulty in defining metrics, existing works define bias loosely as demographic inequality and use intermediate proxy metrics to comparatively measure bias. Examples include: • Regard Ratio: negative-neutral-positive regard score ratios of text generated from bias-inducing prompts (Sheng et al, 2019) • Sentiment Ratio: negative-neutral-positive sentiment score ratios of text generated from African American English (AAE) versus White-Aligned English (WAE) prompts (Groenwold et al, 2020) • Individual and Group Fairness through Sentiment: comparisons of the sentiment distributions of generated text across demographics and prompts (Huang et al, 2020) • Gendered Word Co-occurrence Score: mean and standard deviations of the absolute log ratio of probabilities: P(word|female terms) to P(word|male terms) across all words in generated text (Bordia and Bowman, 2019) There are also metrics for other bias evaluation setups in continuation generation tasks involving sentiment (Shwartz et al, 2020), the ratio of gendered words (Solaiman et al, 2019;Vig et al, 2020;Dinan et al, 2020a), and other novel metrics Yeo and Chen, 2020). Studies of biases in transformation generation tasks favor metrics of accuracy in terms of successfully transforming text to have a desired property.…”

“…Most existing works define bias metrics through the first association-these biases are relatively easier to analyze, since both the demographic and the textual signals of bias are encapsulated within the text. There are also works that define biases towards people who produce the text (Groenwold et al, 2020) or people to whom the text is addressed (Sheng et al, 2021b), though there are relatively fewer works that study these latter associations.…”

“…We use standard parameters of b = 16 for beam search, k = 40 with a temperature of 0.7 for top-k sampling, and p = 0.95 for nucleus sampling (Holtzman et al, 2019). In terms of bias metrics, we use existing NLG bias metrics: regard ratio (Sheng et al, 2019), sentiment ratio (Groenwold et al, 2020), individual and group fairness through sentiment (IF/GF) (Huang et al, 2020), and a gendered word co-occurrence scores (Bordia and Bowman, 2019). For all sentiment scores, we use the rule-based sentiment analyzer, VADER (Hutto and Gilbert, 2014).…”

Translational NLP: A New Paradigm and General Principles for Natural Language Processing Research

“…This information identifies the sublanguage(s) of interest (Grishman and Kittredge, 1986), which determine the availability and development of appropriate NLP tools (Grishman, 2001). Corporate disclosures, financial news reports, and tweets all require different processing strategies (Xing et al, 2018), as do tweets written by different communities (Blodgett et al, 2016;Groenwold et al, 2020). Ex.…”