The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations

Abstract

This paper presents the results of the RepEval 2017 Shared Task, which evaluated neural network sentence representation learning models on the MultiGenre Natural Language Inference corpus (MultiNLI) recently introduced by Williams et al. (2017). All of the five participating teams beat the bidirectional LSTM (BiLSTM) and continuous bag of words baselines reported in Williams et al.. The best single model used stacked BiLSTMs with residual connections to extract sentence features and reached 74.5% accuracy on the genre-matched test set. Surprisingly, the results of the competition were fairly consistent across the genrematched and genre-mismatched test sets, and across subsets of the test data representing a variety of linguistic phenomena, suggesting that all of the submitted systems learned reasonably domainindependent representations for sentence meaning.

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

“…These examples are included in the distributed corpus, but are marked with '-' in the gold label field, and should not be used in standard evaluations. (Nangia et al, 2017).…”

Character-level Intra Attention Network for Natural Language Inference

“…The baseline model we used here is introduced by (Williams et al, 2017) accompanied with the publication of MNLI corpus. It has a 5-layer structure which is shown in Figure 1.…”

“…We evaluated our approach on the Multi-Genre NLI (MNLI) corpus, as a shared task for RepEval 2017 workshop (Nangia et al, 2017). We train our CIAN model on a mixture of MNLI and SNLI corpus, by using a full MNLI training set and a randomly selected 20 percent of the SNLI training set at each epoch.…”

“…However, most of the sentences in SNLI corpus are short and simple, which limit the room for fine-grained comparisons between models. Currently, a more comprehensive Multi-Genre NLI corpus (MNLI; Williams et al, 2017) of 433K sentence pairs was released, aiming at evaluating large-scale NLI models. Authors gave out some baseline results accompanied by the publish of MNLI corpus, the BiLSTM model achieves an accuracy of 67.5, and the Enhanced Sequential Inference Model (Chen et al, 2016) achieves an accuracy of 72.4.…”

“…In this paper, we use the BiLSTM model from (Williams et al, 2017) as the baseline model for the evaluation of the MNLI corpus. To augment the baseline model, firstly, a character-level convolutional neural network (CharCNN; Kim et al, 2016) is applied.…”