BPE-Dropout: Simple and Effective Subword Regularization

Provilkov, Ivan; Emelianenko, Dmitrii; Voita, Elena

doi:10.48550/arxiv.1910.13267

Cited by 14 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is contrary to the usual tradeoff between the two terms. We speculate that for smaller values of β, the noise from the relaxation causes the optimizer to reduce codebook usage toward the beginning of training, resulting in worse ELB at convergence.5 During training, we apply 10% BPE dropout(Provilkov et al, 2019), whose use is common in the neural machine translation literature.6 Strictly speaking, Equation 1 requires us to sample from the categorical distribution specified by the dVAE encoder logits, rather than taking the argmax. In preliminary experiments on ImageNet, we found that this was a useful regularizer in the overparameterized regime, and allows the transformer to be trained using soft targets for the cross-entropy loss.…”

mentioning

confidence: 99%

Zero-Shot Text-to-Image Generation

Ramesh,

Pavlov,

Goh

et al. 2021

Preprint

271

409

View full text Add to dashboard Cite

Text-to-image generation has traditionally focused on finding better modeling assumptions for training on a fixed dataset. These assumptions might involve complex architectures, auxiliary losses, or side information such as object part labels or segmentation masks supplied during training. We describe a simple approach for this task based on a transformer that autoregressively models the text and image tokens as a single stream of data. With sufficient data and scale, our approach is competitive with previous domain-specific models when evaluated in a zero-shot fashion.

show abstract

mentioning

confidence: 99%

Zero-Shot Text-to-Image Generation

Ramesh,

Pavlov,

Goh

et al. 2021

Preprint

271

409

View full text Add to dashboard Cite

show abstract

“…For the En ↔ Ru Biomedical translation task, we learn a separate BPE tokenizer solely on our Biomedical Task Data described in 2.2. We use BPE-dropout (Provilkov et al, 2019) of 0.1 for both language pairs and tasks. We post-process En → De model generated translations to replace quotes with their German equivalents -" and ".…”

Section: Data Pre-processing and Post-processingmentioning

confidence: 99%

NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21

Subramanian¹,

Hrinchuk²,

Adams³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper provides an overview of NVIDIA NeMo's neural machine translation systems for the constrained data track of the WMT21 News and Biomedical Shared Translation Tasks. Our news task submissions for English ↔ German (En ↔ De) and English ↔ Russian (En ↔ Ru) are built on top of a baseline transformer-based sequence-to-sequence model (Vaswani et al., 2017). Specifically, we use a combination of 1) checkpoint averaging 2) model scaling 3) data augmentation with backtranslation and knowledge distillation from right-to-left factorized models 4) finetuning on test sets from previous years 5) model ensembling 6) shallow fusion decoding with transformer language models and 7) noisy channel re-ranking. Additionally, our biomedical task submission for English ↔ Russian uses a biomedically biased vocabulary and is trained from scratch on news task data, medically relevant text curated from the news task dataset, and biomedical data provided by the shared task. Our news system achieves a sacre-BLEU score of 39.5 on the WMT'20 En → De test set outperforming the best submission from last year's task of 38.8. Our biomedical task Ru → En and En → Ru systems reach BLEU scores of 43.8 and 40.3 respectively on the WMT'20 Biomedical Task Test set, outperforming the previous year's best submissions.

show abstract

“…2. Stochastic algorithms (Kudo, 2018;Provilkov et al, 2019) draw multiple segmentations for source and target sequences resorting to randomization to improve robustness and generalization of translation models. 3.…”

Section: Introductionmentioning

confidence: 99%

“…The performance of a standard subword transformer (Vaswani et al, 2017) trained on WMT datasets tokenized using DPE is compared against Byte Pair Encoding (BPE) (Sennrich et al, 2016) and BPE dropout (Provilkov et al, 2019). Empirical results on English ↔ (German, Romanian, Estonian, Finnish, Hungarian) suggest that stochastic subword segmentation is effective for tokenizing source sentences, whereas deterministic DPE is superior for segmenting target sentences.…”

Section: Introductionmentioning

confidence: 99%

“…Empirical results on machine translation suggest that DPE is effective for segmenting output sentences and can be combined with BPE dropout for stochastic segmentation of source sentences. DPE achieves an average improvement of 0.9 BLEU over BPE (Sennrich et al, 2016) and an average improvement of 0.55 BLEU over BPE dropout (Provilkov et al, 2019) on several WMT datasets including English ↔ (German,…”

mentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation

Haffari

Norouzi

2020

Preprint

View full text Add to dashboard Cite

This paper introduces Dynamic Programming Encoding (DPE), a new segmentation algorithm for tokenizing sentences into subword units. We view the subword segmentation of output sentences as a latent variable that should be marginalized out for learning and inference. A mixed character-subword transformer is proposed, which enables exact log marginal likelihood estimation and exact MAP inference to find target segmentations with maximum posterior probability. DPE uses a lightweight mixed character-subword transformer as a means of pre-processing parallel data to segment output sentences using dynamic programming. Empirical results on machine translation suggest that DPE is effective for segmenting output sentences and can be combined with BPE dropout for stochastic segmentation of source sentences. DPE achieves an average improvement of 0.9 BLEU over BPE (Sennrich et al., 2016) and an average improvement of 0.55 BLEU over BPE dropout (Provilkov et al., 2019) on several WMT datasets including English ↔ (German,

show abstract

BPE-Dropout: Simple and Effective Subword Regularization

Cited by 14 publications

References 0 publications

Zero-Shot Text-to-Image Generation

Zero-Shot Text-to-Image Generation

NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21

Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation

Contact Info

Product

Resources

About