NLITrans at SemEval-2018 Task 12: Transfer of Semantic Knowledge for Argument Comprehension

Abstract

The Argument Reasoning Comprehension Task requires significant language understanding and complex reasoning over world knowledge. We focus on transfer of a sentence encoder to bootstrap more complicated models given the small size of the dataset. Our best model uses a pre-trained BiLSTM to encode input sentences, learns task-specific features for the argument and warrants, then performs independent argument-warrant matching. This model achieves mean test set accuracy of 64.43%. Encoder transfer yields a significant gain to our best model over random initialization. Independent warrant matching effectively doubles the size of the dataset and provides additional regularization. We demonstrate that regularization comes from ignoring statistical correlations between warrant features and position. We also report an experiment with our best model that only matches warrants to reasons, ignoring claims. Relatively low performance degradation suggests that our model is not necessarily learning the intended task.

Probing Neural Network Comprehension of Natural Language Arguments

“…Similar to our experiment with warrants, Poliak et al (2018) classified NLI data based on the hypothesis only. A similar experiment to our probing task was performed by Niven and Kao (2018), but only with reasons and warrants. They found that independent warrant classification with shared parameters provides some regularization against warrant-label cues (Niven and Kao, 2018).…”

“…A similar experiment to our probing task was performed by Niven and Kao (2018), but only with reasons and warrants. They found that independent warrant classification with shared parameters provides some regularization against warrant-label cues (Niven and Kao, 2018). However, this does not solve the problem since the presence of a cue is enough to increase the logits for either warrant.…”

“…The most successful previous work on ARCT (Choi and Lee, 2018;Zhao et al, 2018;Niven and Kao, 2018) involved transfer learning from Natural Language Inference (NLI) datasets (Bowman et al, 2015;Williams et al, 2017), and utilized effective NLI models such as ESIM (Chen et al, 2016) and InferSent (Conneau et al, 2017). More recently, Botschen et al (2018) added FrameNet knowledge with modest performance gains.…”

Shortcut Learning Explanations for Deep Natural Language Processing: A Survey on Dataset Biases

“…Notably, when confronted with shifts in input distribution, such as those encountered during domain adaptation, with out-ofdistribution examples, or when faced with novel scenarios, LLMs exhibit challenges in adapting and providing reliable predictions. This suggests that LLMs may encounter difficulties in adjusting to new input distributions, potentially relying on biases ingrained during training [21] , [22], [23]. One type of bias observed is the strong co-occurrence correlation between certain class labels and specific lexical features.…”