SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup

Abstract

Active learning is an important technique for low-resource sequence labeling tasks. However, current active sequence labeling methods use the queried samples alone in each iteration, which is an inefficient way of leveraging human annotations. We propose a simple but effective data augmentation method to improve label efficiency of active sequence labeling. Our method, SeqMix, simply augments the queried samples by generating extra labeled sequences in each iteration. The key difficulty is to generate plausible sequences along with token-level labels. In SeqMix, we address this challenge by performing mixup for both sequences and token-level labels of the queried samples. Furthermore, we design a discriminator during sequence mixup, which judges whether the generated sequences are plausible or not. Our experiments on Named Entity Recognition and Event Detection tasks show that SeqMix can improve the standard active sequence labeling method by 2.27%-3.75% in terms of F 1 scores. The code and data for SeqMix can be found at https://github. com/rz-zhang/SeqMix.

To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLP

“…Despite their success in text classification and sequence-to-sequence tasks, they are seldom used for sequence tagging tasks. Zhang, Yu, and Zhang (2020) use the mixup technique (Zhang et al 2018) in scope of active learning, where they augment the queries at each iteration and later classify whether the augmented query is plausible-since the resulting queries might be noisy-and report improvements for the Named Entity Recognition (NER) and event detection task. However building a robust discriminator for more challenging tasks as dependency parsing (DP) and semantic role labeling (SRL) is a challenge on its own.…”

“…On a "phrase-level attack", the authors first choose two subtrees and then maximize the error rate on the target subtree by modifying the tokens in the source subtree. Even though the adversarial example generation techniques (Zheng et al 2020;Han et al 2020) could be used to augment data in theory, the requirements such as a separate seq2seq generator, a BERT based scorer (Zhang et al 2020), reference parsers that are of certain quality, external POS taggers and high quality pretrained BERT (Devlin et al 2019) models, make them challenging to apply on low-resource languages. Besides, most of the aforementioned adversarial attacks are optimized to trigger an undesired change in the output with minimal modifications, while data augmentation is only concerned about increasing the generalization capacity of the model.…”

“…Therefore sophisticated techniques that make use of such models are also left out in this paper. (Jindal et al 2020a) Emb/Hidden classification not suitable SPEECHMIX (Jindal et al 2020b) Emb/Hidden Speech/Audio not suitable MIXTEXT Emb/Hidden classification not suitable SWITCHOUT (Wang et al 2018) Input machine translation not suitable SIGNEDGRAPH (Chen, Ji, and Evans 2020) Input paraphrase not suitable DAGA (Ding et al 2020) Input+Label sequence tagging not suitable SEQMIX (Zhang, Yu, and Zhang 2020) Input+Label active sequence labeling not suitable GECA (Andreas 2020) Input Agnostic not suitable…”

Improving Distinction between ASR Errors and Speech Disfluencies with Feature Space Interpolation

“…[27] constructs a new token sequence by randomly selecting a token from two different sequences at each index. [28] suggests applying mixup in feature space, after an intermediary layer of a pretrained LM. New input data is then generated by reversing the synthetic feature to find the most similar token in the vocabulary.…”

“…While these two works involve interpolating text inputs, our method differs significantly in that we do not directly generate augmented training samples; instead, we utilize mixup as a regularizing layer during the training process. Our method also does not require reversing word-embeddings or discriminative filtering using GPT-2 introduced in [28].…”