Data augmentation in natural language processing: a novel text generation approach for long and short text classifiers

Abstract

In many cases of machine learning, research suggests that the development of training data might have a higher relevance than the choice and modelling of classifiers themselves. Thus, data augmentation methods have been developed to improve classifiers by artificially created training data. In NLP, there is the challenge of establishing universal rules for text transformations which provide new linguistic patterns. In this paper, we present and evaluate a text generation method suitable to increase the performance of classifiers for long and short texts. We achieved promising improvements when evaluating short as well as long text tasks with the enhancement by our text generation method. Especially with regard to small data analytics, additive accuracy gains of up to 15.53% and 3.56% are achieved within a constructed low data regime, compared to the no augmentation baseline and another data augmentation technique. As the current track of these constructed regimes is not universally applicable, we also show major improvements in several real world low data tasks (up to +4.84 F1-score). Since we are evaluating the method from many perspectives (in total 11 datasets), we also observe situations where the method might not be suitable. We discuss implications and patterns for the successful application of our approach on different types of datasets.

Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligence

“…The challenge with using these models is to make the generations truly label preserving. This is, for example, done by Anaby-Tavor et al (2020), Queiroz Abonizio and Barbon Junior (2020) and Bayer et al (2021). The models are conditioned by fine-tuning on the label-induced training data (or just the class data) and are then tasked to complete a text given the label conditioned beginning (prompt).…”

“…Our data augmentation strategy is the first to explore the generation capabilities of large language models with constraining them through filtering mechanisms. We combine the works of Yoo et al (2021) and Bayer et al (2021) by using GPT-3 with a human-in-the-loop filtering mechanism. We extend the few-shot learning research by proposing a multi-level fine-tuning approach.…”

“…GPT-3 Data Augmentation: With data augmentation we can create new instances from existing ones, which can be particularly advantageous when the amount of data is small. We propose a data augmentation strategy based on text generation with GPT-3 (Brown et al, 2020), which is inspired by the method from Yoo et al (2021) and Bayer et al (2021). GPT-3 can be tasked to complete a given text, also called a prompt.…”

“…This is particularly relevant for urban infrastructures, which require high resilience against cyberattacks, as well as for CERTs, as they need to collect and communicate information in the most reliable and targeted way possible (Riebe et al, 2021a). The data augmentation strategy is inspired by the work of Bayer et al (2021) and follows the example of Yoo et al (2021) by utilizing the large generation model GPT-3 to generate new instances based on the few existing labeled ones.…”

AIGenC: AI generalisation via creativity

“…The problem of extrapolating without additional data has been targeted recently by preliminary approaches such as continual learning for non-stationary data or transfer learning, neither of which has offered fully satisfactory solutions (Mitra, 2021). Some successful systems appear to perform as if they had learnt complex and abstract concepts and could potentially transfer learning (e.g., text to image generation (Ramesh et al, 2021), text generation (Bayer et al, 2022;Yang et al, 2021) etc.). This illusion is nurtured by the assumption that human-like performance implies human-like strategies, that is, by the belief that behaving in a manner akin to humans presupposes similar underlying cognitive traits.…”