Towards Bidirectional Hierarchical Representations for
            Attention-based Neural Machine Translation

Yang, Baosong; Wong, Derek F.; Xiao, Tong; Chao, Lidia S.; Zhu, Jun

doi:10.18653/v1/d17-1150

Cited by 24 publications

(15 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As seen in Figure 4 (b) and (c), the performance of pre-trained NMT encoders obviously became worse on long-distance cases across language pairs and model variants. This is consistent with prior observation on NMT systems that both RNN and SAN fail to fully capture long-distance dependencies (Tai et al, 2015;Yang et al, 2017;Tang et al, 2018).…”

Section: Discussionsupporting

confidence: 92%

Assessing the Ability of Self-Attention Networks to Learn Word Order

Yang

Wang

Wong

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Self-attention networks (SAN) have attracted a lot of interests due to their high parallelization and strong performance on a variety of NLP tasks, e.g. machine translation. Due to the lack of recurrence structure such as recurrent neural networks (RNN), SAN is ascribed to be weak at learning positional information of words for sequence modeling. However, neither this speculation has been empirically confirmed, nor explanations for their strong performances on machine translation tasks when "lacking positional information" have been explored. To this end, we propose a novel word reordering detection task to quantify how well the word order information learned by SAN and RNN. Specifically, we randomly move one word to another position, and examine whether a trained model can detect both the original and inserted positions. Experimental results reveal that: 1) SAN trained on word reordering detection indeed has difficulty learning the positional information even with the position embedding; and 2) SAN trained on machine translation learns better positional information than its RNN counterpart, in which position embedding plays a critical role. Although recurrence structure make the model more universally-effective on learning word order, learning objectives matter more in the downstream tasks such as machine translation.

show abstract

Section: Discussionsupporting

confidence: 92%

Assessing the Ability of Self-Attention Networks to Learn Word Order

Yang

Wang

Wong

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Therefore, we merely apply locality modeling to the lower layers, which same to the configuration in Yu et al (2018) and . In this way, the representations are learned in a hierarchical fashion (Yang et al, 2017). That is, the distance-aware and local information extracted by the lower SAN layers, is expected to complement distance-agnostic and global information captured by the higher SAN layers.…”

Section: Locality Modeling Via 1d Convolutionmentioning

confidence: 99%

Convolutional Self-Attention Networks

Yang¹,

Wang²,

Wong³

et al. 2019

Proceedings of the 2019 Conference of the North

Self Cite

107

View full text Add to dashboard Cite

Self-attention networks (SANs) have drawn increasing interest due to their high parallelization in computation and flexibility in modeling dependencies. SANs can be further enhanced with multi-head attention by allowing the model to attend to information from different representation subspaces. In this work, we propose novel convolutional self-attention networks, which offer SANs the abilities to 1) strengthen dependencies among neighboring elements, and 2) model the interaction between features extracted by multiple attention heads. Experimental results of machine translation on different language pairs and model settings show that our approach outperforms both the strong Transformer baseline and other existing models on enhancing the locality of SANs. Comparing with prior studies, the proposed model is parameter free in terms of introducing no more parameters.

show abstract

“…We refer to models containing this topdown pass as top-down tree2seq models. Note that such a top-down pass has been shown to aid in tree-based NMT with supervised syntactic information (Chen et al, 2017a;Yang et al, 2017); here, we add it to our unsupervised hierarchies.…”

Section: Top-down Encoder Passmentioning

confidence: 99%

“…Eriguchi et al (2016) introduced a tree-LSTM encoder for NMT that relied on an external parser to parse the training and test data. The tree-LSTM encoder was improved upon by Chen et al (2017a) and Yang et al (2017), who added a top-down pass. Other approaches have used convolutional networks to model source syntax.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Source Hierarchies for Low-Resource Neural Machine Translation

Currey¹,

Heafield²

2018

Proceedings of the Workshop on the Relevance of Linguistic Structure in Neural Architectures for NLP

View full text Add to dashboard Cite

Incorporating source syntactic information into neural machine translation (NMT) has recently proven successful (Eriguchi et al., 2016; Luong et al., 2016). However, this is generally done using an outside parser to syntactically annotate the training data, making this technique difficult to use for languages or domains for which a reliable parser is not available. In this paper, we introduce an unsupervised tree-to-sequence (tree2seq) model for neural machine translation; this model is able to induce an unsupervised hierarchical structure on the source sentence based on the downstream task of neural machine translation. We adapt the Gumbel tree-LSTM of Choi et al. (2018) to NMT in order to create the encoder. We evaluate our model against sequential and supervised parsing baselines on three low-and medium-resource language pairs. For low-resource cases, the unsupervised tree2seq encoder significantly outperforms the baselines; no improvements are seen for medium-resource translation.

show abstract

Towards Bidirectional Hierarchical Representations for Attention-based Neural Machine Translation

Cited by 24 publications

References 17 publications

Assessing the Ability of Self-Attention Networks to Learn Word Order

Assessing the Ability of Self-Attention Networks to Learn Word Order

Convolutional Self-Attention Networks

Unsupervised Source Hierarchies for Low-Resource Neural Machine Translation

Contact Info

Product

Resources

About