Tensor2Tensor for Neural Machine Translation

Abstract

Tensor2Tensor is a library for deep learning models that is well-suited for neural machine translation and includes the reference implementation of the state-of-the-art Transformer model. Neural Machine Translation BackgroundMachine translation using deep neural networks achieved great success with sequence-tosequence models (Sutskever et al., 2014;Bahdanau et al., 2014;Cho et al., 2014) that used recurrent neural networks (RNNs) with LSTM cells (Hochreiter and Schmidhuber, 1997). The basic sequence-to-sequence architecture is composed of an RNN encoder which reads the source sentence one token at a time and transforms it into a fixed-sized state vector. This is followed by an RNN decoder, which generates the target sentence, one token at a time, from the state vector.While a pure sequence-to-sequence recurrent neural network can already obtain good translation results (Sutskever et al., 2014;Cho et al., 2014), it suffers from the fact that the whole input sentence needs to be encoded into a single fixed-size vector. This clearly manifests itself in the degradation of translation quality on longer sentences and was partially overcome in Bahdanau et al. (2014) by using a neural model of attention.Convolutional architectures have been used to obtain good results in word-level neural machine translation starting from Kalchbrenner and Blunsom (2013) and later in Meng et al. (2015). These early models used a standard RNN on top of the convolution to generate the output, which creates a bottleneck and hurts performance.Fully convolutional neural machine translation without this bottleneck was first achieved in Kaiser and Bengio (2016) and Kalchbrenner et al. (2016). The Extended Neural GPU model (Kaiser and Bengio, 2016) used a recurrent stack of gated convolutional layers, while the ByteNet model (Kalchbrenner et al., 2016) did away with recursion and used left-padded convolutions in the decoder. This idea, introduced in WaveNet (van den Oord et al., 2016), significantly improves efficiency of the model. The same technique was improved in a number of neural translation models recently, including Gehring et al. (2017) and Kaiser et al. (2017). Self-AttentionInstead of convolutions, one can use stacked self-attention layers. This was introduced in the Transformer model (Vaswani et al., 2017) and has significantly improved state-of-the-art in machine translation and language modeling while also improving the speed of training. Research

Billion-scale Pre-trained E-commerce Product Knowledge Graph Model

“…2) Model: Base model Paraphrase identification is a wellstudied sentence pair modeling task and is very useful for many NLP applications such as machine translation(MT) [14], question answering(QA) [15] and information retrieval(IR) [16]. Many methods have been proposed for it in recent years including pairwise word interaction modeling with deep neural network system [17], character level neural network model [18], and pre-trained language model [2].…”

Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition

“…For all investigated approaches including non-fusion baselines Baseline-mag and Baseline-phase, we use beam-search decoding during inference and slightly deviate from the standard transformer architecture in [6] by using layer normalization before each attention or stack of fully connected layers according to the implementation in [33]. During decoding, the final output P final ℓ of all approaches can optionally be computed as…”

ComFormer: Code Comment Generation via Transformer and Fusion Method-based Hybrid Code Representation

“…In this study, we propose a novel method ComFormer via Transformer [13] and fusion method-based hybrid code representation. Our method considers Transformer since this deep learning model can achieve better performance than traditional sequence to sequence models in classical natural language processing (NLP) tasks (such as neural machine translation [14] [15] and software engineering [16]). Moreover, our method also utilizes the hybrid code representation to effectively learn the semantic of the code since this representation can extract both lexical-level and syntactic-level information from the code, respectively.…”