Massive Exploration of Neural Machine Translation Architectures

Abstract

Neural Machine Translation (NMT) has shown remarkable progress over the past few years, with production systems now being deployed to end-users. As the field is moving rapidly, it has become unclear which elements of NMT architectures have a significant impact on translation quality. In this work, we present a large-scale analysis of the sensitivity of NMT architectures to common hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on a WMT English to German translation task. Our experiments provide practical insights into the relative importance of factors such as embedding size, network depth, RNN cell type, residual connections, attention mechanism, and decoding heuristics. As part of this contribution, we also release an open-source NMT framework in TensorFlow to make it easy for others to reproduce our results and perform their own experiments.

A surprisingly effective out-of-the-box char2char model on the E2E NLG Challenge dataset

“…While in their large-scale NMT experiments (Britz et al, 2017) use word-based sequences, in our case we use character-based ones. This simply involves changing "delimiter" option in configuration files.…”

“…Contrary to word-based sequences, we use character-based sequences for generating grammatically correct and natural utterances. (Britz et al, 2017), provides an overview of the framework. While many options are configurable (number of layers, unidirectional vs bidirectional encoder, additive vs multiplicative attention mechanism, GRU (Cho et al, 2014) vs LSTM cells (Hochreiter and Schmidhuber, 1997), etc.…”

When does a compliment become sexist? Analysis and classification of ambivalent sexism using twitter data

“…The implementation of the described Sequence to Sequence model has been done using tf-seq2seq framework (Britz et al, 2017) for Tensorflow (Abadi et al, 2016). The experiment was conducted after splitting the training set and the test set in the ratio 7 : 3.…”

Proceedings of the First Workshop on Neural Machine Translation

“…Recently, Britz et al (2017) have released a paper about exploring the hyper-parameters of NMT. This work is similar to our paper in the terms of finding the better hyper-parameters by doing a large number of experiments and deriving empirical conclusions.…”

“…While this can lead to much faster convergence, the resulting models are shown to slightly underperform compared to annealing SGD . However, Adam's speed and reputation of generally being "good enough" have made it a popular choice for researchers and NMT toolkit authors 6 (Arthur et al, 2016;Lee et al, 2016;Britz et al, 2017;Sennrich et al, 2017).…”