Forest-Based Neural Machine Translation

Ma, Chaoyong; Tamura, Akihiro; Utiyama, Masao; Zhao, Tiejun; Sumita, Eiichiro

doi:10.18653/v1/p18-1116

Cited by 21 publications

(18 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although Ma et al (2018) concludes that for RNN-based NMT systems, using packed forests is definitely better than using constituent trees, we found that it is not necessarily true for our syntax-based Transformer. For some configurations, using forests performs worse than using trees.…”

Section: Tree or Forest?contrasting

confidence: 61%

“…Note that we used absolute positional encoding instead of syntax-based positional encoding, so we can conclude that the improvement on performance was completely from syntax scores. This indicates that scores are important for the syntax-based Transformer, as was the case with the syntax-based NMT based on RNNs as demonstrated in Ma et al (2018).…”

Section: Influence Of Scoresmentioning

confidence: 73%

“…Despite the relatively worse BLEU score on En-to-Ja task, our system is much faster. On En-to-Ja tasks, it takes about 22 hours with one GPU to train our model, while for the system of Ma et al (2018), the training time is about 240 hours (11 times slower than ours).…”

Section: Resultsmentioning

confidence: 89%

“…Furthermore, as demonstrated by Ma et al (2018), scores are important for the performance of syntax-based NMT systems. To utilize the scores, we calculate the attention by Equation 20rather than Equation 2, which is similar to the Score-on-Attention model of Ma et al (2018).…”

Section: Syntax-based Self-attentionmentioning

confidence: 99%

“…Syntactic information can be used on either the source-side (Eriguchi, Tsuruoka, and Cho 2017), or the target-side (Aharoni and Goldberg 2017), or both (Wu, Zhang, Zhang, Yang, Li, and Zhou 2018). Syntactic information can be represented as constituent trees (Eriguchi, Hashimoto, and Tsuruoka 2016), packed forests (Ma, Tamura, Utiyama, Zhao, and Sumita 2018), or graphs (Hashimoto and Tsuruoka 2017). There are also some attempts at using syntactic information explicitly in Transformer (Strubell, Verga, Andor, Weiss, and McCallum 2018).…”

mentioning

confidence: 99%

See 4 more Smart Citations

Syntax-based Transformer for Neural Machine Translation

Tamura

Utiyama

et al. 2020

Journal of Natural Language Processing

Self Cite

View full text Add to dashboard Cite

Polosukhin 2017), which purely depends on attention mechanism, has achieved stateof-the-art performance on machine translation (MT). However, syntactic information, which has improved many previous MT models, has not been utilized explicitly by Transformer. We propose a syntax-based Transformer for MT, which incorporates source-side syntax structures generated by the parser into the self-attention and positional encoding of the encoder. Our method is general in that it is applicable to both constituent trees and packed forests. Evaluations on two language pairs show that our syntax-based Transformer outperforms the conventional (non-syntactic) Transformer. The improvements of BLEUs on English-Japanese, English-Chinese and English-German translation tasks are up to 2.32, 2.91 and 1.03, respectively. Furthermore, our ablation study and qualitative analysis demonstrate that the syntax-based self-attention does well in learning local structural information, while the syntax-based positional encoding does well in learning global structural information.

show abstract

Section: Tree or Forest?contrasting

confidence: 61%

Section: Influence Of Scoresmentioning

confidence: 73%

Section: Resultsmentioning

confidence: 89%

Section: Syntax-based Self-attentionmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Syntax-based Transformer for Neural Machine Translation

Tamura

Utiyama

et al. 2020

Journal of Natural Language Processing

Self Cite

View full text Add to dashboard Cite

show abstract

Entity-Aware Biaffine Attention for Constituent Parsing

Xinyi

Yin

Zhang

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Constituency parsing involves analyzing a sentence by breaking it into subphrases, or constituents. While many deep neural models have achieved stateof-the-art performance in this task, they often overlook the entity-violating issue, where an entity fails to form a complete sub-tree in the resultant parsing tree. To address this, we propose an entity-aware biaffine attention model for constituent parsing. This model incorporates entity information into the biaffine attention mechanism by using additional entity role vectors for potential phrases, which enhances the parsing accuracy. We introduce a new metric, the Entity Violating Rate (EVR), to quantify the extent of entity violations in parsing results. Experiments on three popular datasets-ONTONOTES, PTB, and CTB-demonstrate that our model achieves the lowest EVR while maintaining high precision, recall, and F1scores comparable to existing models. Further evaluation in downstream tasks, such as sentence sentiment analysis, highlights the effectiveness of our model and the validity of the proposed EVR metric.

show abstract