Self-Supervised Contextual Data Augmentation for Natural Language Processing

Abstract

In this paper, we propose a novel data augmentation method with respect to the target context of the data via self-supervised learning. Instead of looking for the exact synonyms of masked words, the proposed method finds words that can replace the original words considering the context. For self-supervised learning, we can employ the masked language model (MLM), which masks a specific word within a sentence and obtains the original word. The MLM learns the context of a sentence through asymmetrical inputs and outputs. However, without using the existing MLM, we propose a label-masked language model (LMLM) that can include label information for the mask tokens used in the MLM to effectively use the MLM in data with label information. The augmentation method performs self-supervised learning using LMLM and then implements data augmentation through the trained model. We demonstrate that our proposed method improves the classification accuracy of recurrent neural networks and convolutional neural network-based classifiers through several experiments for text classification benchmark datasets, including the Stanford Sentiment Treebank-5 (SST5), the Stanford Sentiment Treebank-2 (SST2), the subjectivity (Subj), the Multi-Perspective Question Answering (MPQA), the Movie Reviews (MR), and the Text Retrieval Conference (TREC) datasets. In addition, since the proposed method does not use external data, it can eliminate the time spent collecting external data, or pre-training using external data.

Consequential Advancements of Self-Supervised Learning (SSL) in Deep Learning Contexts

“…Their study aimed to improve the training efficiency of this model to help handle the precise classification of NLP problems. Park and Ahn (2019) [100] inspected the vital gains of SSL to lead to efficient detection of NLP. Researchers proposed a new approach dedicated to data augmentation that considers the intended context of the data.…”

Search Engine for Open Geospatial Consortium Web Services Improving Discoverability through Natural Language Processing-Based Processing and Ranking

“…Some approaches address the issue by integrating a semantic search capability in the OGC Catalogue Service for Web (CSW) or extracting and validating new dependencies, and, in this way, allowing for a more semantically enriched geospatial data discovery through Web ontology Language (OWL) [5,23,24]. With the introduction of deep learning and language models, some works have applied these in Natural Language Processing to extract valuable insight from unstructured text [25,26]. By leveraging NLP, keywords, textual annotations, and geospatial information within the text can be identified and semantically enriched.…”

A novel textual data augmentation method for identifying comparative text from user-generated content

“…For example, replacing the abbreviated form with the full written form of the phrase (Coulombe, 2018), such as "He's" and 'He is'. Word replacement method represents replacing some randomly selected words with their substitute words having similar context (Abdurrahman and Purwarianti, 2019;Park and Ahn, 2019;Rizos et al, 2019;Wei and Zou, 2019). With back-translation method, an original text is translated into other, intermediate languages, which are then re-translated into the original language (Li et al, 2018a).…”