Word-level Perturbation Considering Word Length and Compositional Subwords

Hiraoka, Tatsuya; Takase, Sho; Uchiumi, Kei; Keyaki, Atsushi; Okazaki, Naoaki

doi:10.18653/v1/2022.findings-acl.258

Cited by 2 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this study, we proposed the inference strategy to mitigate the discrepancy between the training and inference in subword regularizations. In experiments, we focused the subword regularization proposed by Kudo (2018) but we can apply the proposed inference strategy to variants of the subword regularization such as BPE dropout (Provilkov et al, 2020) and compositional word replacement (Hiraoka et al, 2022). Takase and Kiyono (2021) reported that simple perturbations such as word dropout are effective in a large amount of training data.…”

Section: Related Workmentioning

confidence: 99%

Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation

Takase¹,

Hiraoka²,

Okazaki³

2022

Findings of the Association for Computational Linguistics: ACL 2022

Self Cite

View full text Add to dashboard Cite

Subword regularizations use multiple subword segmentations during training to improve the robustness of neural machine translation models. In previous subword regularizations, we use multiple segmentations in the training process but use only one segmentation in the inference. In this study, we propose an inference strategy to address this discrepancy. The proposed strategy approximates the marginalized likelihood by using multiple segmentations including the most plausible segmentation and several sampled segmentations. Because the proposed strategy aggregates predictions from several segmentations, we can regard it as a single model ensemble that does not require any additional cost for training. Experimental results show that the proposed strategy improves the performance of models trained with subword regularization in low-resource machine translation tasks.

show abstract