It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations

Abstract

Training on only perfect Standard English corpora predisposes pre-trained neural networks to discriminate against minorities from nonstandard linguistic backgrounds (e.g., African American Vernacular English, Colloquial Singapore English, etc.). We perturb the inflectional morphology of words to craft plausible and semantically similar adversarial examples that expose these biases in popular NLP models, e.g., BERT and Transformer, and show that adversarially fine-tuning them for a single epoch significantly improves robustness without sacrificing performance on clean data. 1

Explainable AI: A Review of Machine Learning Interpretability Methods

“…The BERT language model was utilised in two studies in order to create textual adversarial examples: Garg and Ramakrishnan [ 162 ] and Li et al [ 163 ], both of which proposed generating adversarial examples through text perturbations that are based on the BERT masked language model, as part of the original text is masked and alternative text pieces are generated to replace these masks. In their work [ 164 ], Tan et al proposed Morpheus, which is a method for generating textual adversarial examples by greedily perturbing the inflections of the original words in the text to find the inflected forms with the greatest loss increase, only taking into considerations the inflections that belong to the same part of speech as the original word. Unlike most work on textual adversarial examples, Morpheus produces its adversaries by exploiting the morphology of the text.…”

Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding

“…However, this generally involves retraining the model with the adversarial data, which is computationally expensive and time-consuming. Tan et al (2020) showed that simply fine-tuning a trained model for a single epoch on appropriately generated adversarial training data is sufficient to harden the model against inflectional adversaries. Instead of adversarial training, Piktus et al (2019) train word embeddings to be robust to misspellings, while Zhou et al (2019b) propose using a BERT-based model to detect adversaries and recover clean examples.…”

“…Large-scale neural models have proven successful at a wide range of natural language processing (NLP) tasks but are susceptible to amplifying discrimination against minority linguistic communities (Hovy and Spruit, 2016;Tan et al, 2020) due to selection bias in the training data and model overamplification (Shah et al, 2019).…”

“…In particular, Tan et al (2020) show that current question answering and machine translation systems are overly sensitive to non-standard inflections-a common feature of dialects such as Colloquial Singapore English (CSE) and African American Vernacular English (AAVE). 2 Since people naturally correct for or ignore non-standard inflection use (Foster and Wigglesworth, 2016), we should expect NLP systems to be equally robust.…”

“…Existing work on adversarial robustness for NLP primarily focuses on adversarial training methods (Belinkov and Bisk, 2018;Ribeiro et al, 2018;Tan et al, 2020) or classifying and correcting adversarial examples (Zhou et al, 2019a). However, this effectively increases the size of the training dataset by including adversarial examples or training a new model to identify and correct perturbations, thereby significantly increasing the overall computational cost of creating robust models.…”