IIT-KGP at COIN 2019: Using pre-trained Language Models for modeling Machine Comprehension

Abstract

In this paper, we describe our system for COIN 2019 Shared Task 1: Commonsense Inference in Everyday Narrations Ostermann et al. (2019). We show the power of leveraging state-of-the-art pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers) Devlin et al. (2018) and XLNet Yang et al. (2019) over other Commonsense Knowledge Base Resources such as ConceptNet Speer et al. (2018) and NELL Mitchell et al. (2015) for modeling machine comprehension. We used an ensemble of BERT Large and XLNet Large. Experimental results show that our model gives substantial improvements over the baseline and other systems incorporating knowledge bases and got the 2nd position on the final test set leaderboard with an accuracy of 90.5%.

A Survey on Machine Reading Comprehension Systems

“…In recent years, with the advent of attention-based Transformer architecture (Vaswani et al 2017) as an alternative to common sequential structures like RNN, new Transformer-based language models, such as BERT (Devlin et al 2018) and XLNet (Yang et al 2019b), have been introduced. They are used as the basis for new state-of-the-art results in MRC task (Sharma and Roychowdhury 2019;Su et al 2019;Yang et al 2019a;Tu et al 2020;Zhang et al 2020a;Zhang et al 2020b) by adding or modifying final layers and fine-tuning them on the target task.…”

“…. .. Multiple Choice (Wang et al 2018c), (Greco et al 2016), (Wang et al 2018a), (Lin et al 2018), , (Liu et al 2018a), (Yang et al 2017a), (Yin et al 2016), (Wang et al 2016), (Chaturvedi et al 2018), (Sheng et al 2018), (Sugawara et al 2018), (Tay et al 2018), (Miao et al 2019a), (Chen et al 2019b), (Huang et al 2019b), (Wang et al 2019b), (Sharma and Roychowdhury 2019), (Sun et al 2019), (Xia et al 2019), (Yu et al 2019), (Tu et al 2019), (Tang et al 2019b), (Miao et al 2019b), (Li et al 2019b), (Guo et al 2020), (Niu et al 2020), (Zhou et al 2020b), (Zhang et al 2020b), (Zhang et al 2020a, (Song et al 2020), (Zhang et al 2020c) . .…”

Commonsense Inference in Natural Language Processing (COIN) - Shared Task Report

“…• IIT-KGP (Sharma and Roychowdhury, 2019) present an ensemble of different pretrained language models, namely BERT and XLNet. Both models are pretrained on the RACE data (Lai et al, 2017), and their output is averaged for a final prediction.…”

Adversarial Training for Commonsense Inference

“…Recent approaches are based on the use of pre-trained Transformer-based language models such as BERT (Devlin et al, 2019). Some approaches rely solely on these models by adopting either a single or multi-stage fine-tuning approach (by fine-tuning using additional datasets in a stepwise manner) (Li and Xie, 2019;Sharma and Roychowdhury, 2019;Liu and Yu, 2019; Paragraph: It's a very humbling experience when you need someone to dress you every morning, tie your shoes, and put your hair up. Every menial task takes an unprecedented amount of effort.…”