Semantic Sentence Matching with Densely-Connected Recurrent and Co-Attentive Information

Abstract

Sentence matching is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. For these tasks, understanding logical and semantic relationship between two sentences is required but it is yet challenging. Although attention mechanism is useful to capture the semantic relationship and to properly align the elements of two sentences, previous methods of attention mechanism simply use a summation operation which does not retain original features enough. Inspired by DenseNet, a densely connected convolutional network, we propose a densely-connected co-attentive recurrent neural network, each layer of which uses concatenated information of attentive features as well as hidden features of all the preceding recurrent layers. It enables preserving the original and the co-attentive feature information from the bottommost word embedding layer to the uppermost recurrent layer. To alleviate the problem of an ever-increasing size of feature vectors due to dense concatenation operations, we also propose to use an autoencoder after dense concatenation. We evaluate our proposed architecture on highly competitive benchmark datasets related to sentence matching. Experimental results show that our architecture, which retains recurrent and attentive features, achieves state-of-the-art performances for most of the tasks.

Simple and Effective Text Matching with Richer Alignment Features

“…We compare single models and ensemble models. For a fair comparison, we only compare with results obtained without external contextualized embed- (Chen et al, 2017) 4.3M 88.0 DIIN (Gong et al, 2018) 4.4M 88.0 MwAN (Tan et al, 2018) 14M 88.3 CAFE (Tay et al, 2018b) 4.7M 88.5 HIM (Chen et al, 2017) 7.7M 88.6 SAN (Liu et al, 2018) 3.5M 88.6 CSRAN (Tay et al, 2018a) 13.9M 88.7 DRCN (Kim et al, 2018) 6.7M 88.9 RE2 ( Model Acc(%) ESIM (Chen et al, 2017) 70.6 DecompAtt (Parikh et al, 2016) 72.3 DGEM (Khot et al, 2018) 77.3 HCRN (Tay et al, 2018c) 80.0 CAFE (Tay et al, 2018b) 83.3 CSRAN (Tay et al, 2018a) 86.7 RE2 (ours) 86.0±0.6 dings. In the ensemble experiment, we train 8 models with different random seeds and ensemble the results by a voting strategy.…”

“…POS tags are found in many previous works including Tay et al (2018b) and Gong et al (2018). The exact match of lemmatized tokens is reported as a powerful binary feature in Gong et al (2018) and Kim et al (2018). The second way is adding complexity to the alignment computation.…”

“…CSRAN (Tay et al, 2018a) performs multi-level attention refinement with dense connections among multiple levels. DRCN (Kim et al, 2018) stacks encoding and alignment layers. It concatenates all previously aligned results and has to use an autoencoder to deal with exploding feature spaces.…”

“…However, suffering from inefficient propagation of lower-level features and vanishing gradients, these deeper architectures are harder to train. Recent works have come up with ways of connecting stacked building blocks including dense connection (Tay et al, 2018a;Kim et al, 2018) and recurrent neural networks (Liu et al, 2018), which strengthen the propagation of lower-level features and yield better results than those with a single alignment process. This paper presents RE2, a fast and strong neural architecture with multiple alignment processes for general purpose text matching.…”

Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction

“…They reach an accuracy of 91.1%. Kim et al (2019) exploit denselyconnected co-attentive recurrent neural network, and reach 90% accuracy. In our scenario, we generate pseudo premises and hypotheses, then apply the standard transformer encoder (Ashish et al, 2017;Devlin et al, 2019) to train two NLI models.…”