Does String-Based Neural MT Learn Source Syntax?

Shi, Xing; Padhi, Inkit; Knight, Kevin

doi:10.18653/v1/d16-1159

Cited by 298 publications

(294 citation statements)

References 17 publications

Supporting

Mentioning

268

Contrasting

Unclassified

Order By: Relevance

“…In Section 4.2, we conduct experiments on the multi-granularity label prediction tasks (Shi et al, 2016), and investigate the representations of NMT encoders trained on both translation data and the training data of the label prediction tasks. Experimental results show that the proposed MG-SA indeed captures useful phrase information at various levels of granularities in both scenarios (Q3).…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Granularity Self-Attention for Neural Machine Translation

Hao¹,

Wang²,

Shi³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Current state-of-the-art neural machine translation (NMT) uses a deep multi-head selfattention network with no explicit phrase information. However, prior work on statistical machine translation has shown that extending the basic translation unit from words to phrases has produced substantial improvements, suggesting the possibility of improving NMT performance from explicit modeling of phrases. In this work, we present multi-granularity self-attention (MG-SA): a neural network that combines multi-head selfattention and phrase modeling. Specifically, we train several attention heads to attend to phrases in either n-gram or syntactic formalism. Moreover, we exploit interactions among phrases to enhance the strength of structure modeling -a commonly-cited weakness of self-attention. Experimental results on WMT14 English-to-German and NIST Chinese-to-English translation tasks show the proposed approach consistently improves performance. Targeted linguistic analysis reveals that MG-SA indeed captures useful phrase information at various levels of granularities.

show abstract

Section: Methodsmentioning

confidence: 99%

“…Encoder Layers Recent works (Shi et al, 2016;Peters et al, 2018) show that different layers in encoder tend to capture different syntax and semantic features. Hence, there may have different needs for modeling phrase structure in each layer.…”

Section: Phrase Compositionmentioning

confidence: 99%

Multi-Granularity Self-Attention for Neural Machine Translation

Hao¹,

Wang²,

Shi³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

show abstract

“…We use the default hyperparameters of de in our experiments. We fine-tune each model by training it further only on the target treebank (Shi et al, 2016). We use early stopping based on Label Attachment Score (LAS) on development set.…”

Section: Parameter Sharingmentioning

confidence: 99%

“…1a. We investigate the extent to which data augmentation is useful for learning POS features, using diagnostic classifiers (Veldhoen et al, 2016;Adi et al, 2016;Shi et al, 2016) tions (i.e., the output of word-level biLSTM, h i in Eq. 2), for the training and development data.…”

Section: Analysis Of Data Augmentationmentioning

confidence: 99%

A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages

Vania¹,

Kementchedjhieva²,

Søgaard³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Parsers are available for only a handful of the world's languages, since they require lots of training data. How far can we get with just a small amount of training data? We systematically compare a set of simple strategies for improving low-resource parsers: data augmentation, which has not been tested before; cross-lingual training; and transliteration. Experimenting on three typologically diverse low-resource languages-North Sámi, Galician, and Kazah-We find that (1) when only the low-resource treebank is available, data augmentation is very helpful; (2) when a related high-resource treebank is available, cross-lingual training is helpful and complements data augmentation; and (3) when the high-resource treebank uses a different writing system, transliteration into a shared orthographic spaces is also very helpful.

show abstract

“…[13] note that their system performs reasonably well in both tagging and parsing. [31] present an in-depth analysis of the syntactic knowledge learned by the recurrent sequence-to-sequence NMT.…”

Section: Related Workmentioning

confidence: 99%

Promoting the Knowledge of Source Syntax in Transformer NMT Is Not Needed

Pham¹,

Macháček²,

Bojar³

2019

CyS

View full text Add to dashboard Cite

The utility of linguistic annotation in neural machine translation seemed to had been established in past papers. The experiments were however limited to recurrent sequence-to-sequence architectures and relatively small data settings.We focus on the state-of-the-art Transformer model and use comparably larger corpora. Specifically, we try to promote the knowledge of source-side syntax using multi-task learning either through simple data manipulation techniques or through a dedicated model component. In particular, we train one of Transformer attention heads to produce source-side dependency tree.Overall, our results cast some doubt on the utility of multi-task setups with linguistic information. The data manipulation techniques, recommended in previous works, prove ineffective in large data settings.The treatment of self-attention as dependencies seems much more promising: it helps in translation and reveals that Transformer model can very easily grasp the syntactic structure. An important but curious result is, however, that identical gains are obtained by using trivial "linear trees" instead of true dependencies. The reason for the gain thus may not be coming from the added linguistic knowledge but from some simpler regularizing effect we induced on self-attention matrices.

show abstract

Does String-Based Neural MT Learn Source Syntax?

Cited by 298 publications

References 17 publications

Multi-Granularity Self-Attention for Neural Machine Translation

Multi-Granularity Self-Attention for Neural Machine Translation

A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages

Promoting the Knowledge of Source Syntax in Transformer NMT Is Not Needed

Contact Info

Product

Resources

About