Proceedings of the Ninth Workshop on Statistical Machine Translation

Bojar, Ondřej; Buck, Christian; Federmann, Christian; Haddow, Barry; Koehn, Philipp; Monz, Christof; Post, Matt; Specia, Lucia

doi:10.3115/v1/w14-33

Cited by 11 publications

(5 citation statements)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to GPT-3, the training data for Base LM is around 90% English and includes some text in other languages that was not specifically used to train the model to perform machine translation. We also evaluate FLAN's performance on machine translation for the three datasets evaluated in the GPT-3 paper: French-English from WMT'14 (Bojar et al, 2014), and German-English and Romanian-English from WMT'16 (Bojar et al, 2016).…”

Section: Translationmentioning

confidence: 99%

Finetuned Language Models Are Zero-Shot Learners

Lee¹,

Bosma²,

Zhao³

et al. 2021

Preprint

183

161

View full text Add to dashboard Cite

This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning-finetuning language models on a collection of tasks described via instructions-substantially boosts zero-shot performance on unseen tasks.We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this instructiontuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 19 of 25 tasks that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number * Lead contributors. Author contributions listed at end of paper.

show abstract

Section: Translationmentioning

confidence: 99%

Finetuned Language Models Are Zero-Shot Learners

Lee¹,

Bosma²,

Zhao³

et al. 2021

Preprint

183

161

View full text Add to dashboard Cite

show abstract

“…We perform our comparison on four datasets of varying sizes: IWSLT14 German-English (Federico et al, 2014), WMT14 English-{German, French} (Bojar et al, 2014), and WMT18 English-Chinese (Bojar et al, 2018). We split each dataset into train and validation pairs and evaluate DEÑEN models on the test sets TED-{dev10,dev12, test10, tst11, tst12}, EN-{DE, FR} models on newstest14 and ENÑZH models on new-stest17.…”

Section: Datasets and Evaluationmentioning

confidence: 99%

Bridging the Gap between Position-Based and Content-Based Self-Attention for Neural Machine Translation

Schmidt,

Di Gangi

2023

Proceedings of the Eighth Conference on Machine Translation

View full text Add to dashboard Cite

Position-based token-mixing approaches, such as FNet and MLPMixer, have shown to be exciting attention alternatives for computer vision and natural language understanding. The motivation is usually to remove redundant operations for higher efficiency on consumer GPUs while maintaining Transformer quality. On the hardware side, research on memristive crossbar arrays shows the possibility of efficiency gains up to two orders of magnitude by performing in-memory computation with weights stored on device. While it is impossible to store dynamic attention weights based on token-token interactions on device, position-based weights represent a concrete alternative if they only lead to minimal degradation. In this paper, we propose position-based attention as a variant of multihead attention where the attention weights are computed from position representations. A naive replacement of token vectors with position vectors in self-attention results in a significant loss in translation quality, which can be recovered by using relative position representations and a gating mechanism. We show analytically that this gating mechanism introduces some form of word dependency and validate its effectiveness experimentally under various conditions. The resulting network, rPosNet, outperforms previous position-based approaches and matches the quality of the Transformer with relative position embedding while requiring 20% less attention parameters after training. 1

show abstract

“…Dataset We test our quantization strategy in 3 different translation directions: English-to-German (En2De), English-to-French (En2Fr), and Englishto-Japanese (En2Jp). For En2De and En2Fr, we utilize all of the trainset of WMT2014 and use newstest2013 as devset and newstest2014 as testset (Bojar et al, 2014). For En2Jp, we use KFTT (Neubig, 2011), JESC (Pryzant et al, 2018), and WIT 3 (Cettolo et al, 2012) corpus.…”

Section: Experimental Settingsmentioning

confidence: 99%

Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation

Chung¹,

Kim²,

Choi³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

The deployment of widely used Transformer architecture is challenging because of heavy computation load and memory overhead during inference, especially when the target device is limited in computational resources such as mobile or edge devices. Quantization is an effective technique to address such challenges. Our analysis shows that for a given number of quantization bits, each block of Transformer contributes to translation quality and inference computations in different manners. Moreover, even inside an embedding block, each word presents vastly different contributions. Correspondingly, we propose a mixed precision quantization strategy to represent Transformer weights by an extremely low number of bits (e.g., under 3 bits). For example, for each word in an embedding block, we assign different quantization bits based on statistical property. Our quantized Transformer model achieves 11.8× smaller model size than the baseline model, with less than -0.5 BLEU. We achieve 8.3× reduction in run-time memory footprints and 3.5× speed up (Galaxy N10+) such that our proposed compression strategy enables efficient implementation for on-device NMT.

show abstract

Proceedings of the Ninth Workshop on Statistical Machine Translation

Cited by 11 publications

References 83 publications

Finetuned Language Models Are Zero-Shot Learners

Finetuned Language Models Are Zero-Shot Learners

Bridging the Gap between Position-Based and Content-Based Self-Attention for Neural Machine Translation

Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation

Contact Info

Product

Resources

About