Multi-view Subword Regularization

Abstract

Multilingual pretrained representations generally rely on subword segmentation algorithms to create a shared multilingual vocabulary. However, standard heuristic algorithms often lead to sub-optimal segmentation, especially for languages with limited amounts of data. In this paper, we take two major steps towards alleviating this problem. First, we demonstrate empirically that applying existing subword regularization methods (Kudo, 2018;Provilkov et al., 2020) during fine-tuning of pre-trained multilingual representations improves the effectiveness of cross-lingual transfer. Second, to take full advantage of different possible input segmentations, we propose Multi-view Subword Regularization (MVR), a method that enforces the consistency between predictions of using inputs tokenized by the standard and probabilistic segmentations. Results on the XTREME multilingual benchmark show that MVR brings consistent improvements of up to 2.5 points over using standard segmentation algorithms. 1

CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

“…Subword regularization (Kudo, 2018) and BPE-dropout (Provilkov et al, 2020) recognize that deterministic segmentation during training limits the ability to leverage morphology and word composition; instead, they sample at random one of the multiple tokenizations of the training input, made possible by the inherent ambiguity of subword vocabularies. Wang et al (2021) recently expanded on this paradigm to enforce consistency of predictions over different segmentations. Unigram LM (Kudo, 2018), a segmentation technique that builds its vocabulary top-down, was shown to align with morphology better than BPE on modern pre-trained encoders (Bostrom and Durrett, 2020).…”

Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noise

“…work with code bases byWolf et al (2020) andWang et al (2021), multilingual BERT (mBERT, Devlin et al (2019)), and the data sets' default splits. Most of the corpora we work with were provided by the Universal Dependency project (UD,Nivre et al (2016)).…”