IIT-KGP at MEDIQA 2019: Recognizing Question Entailment using Sci-BERT stacked with a Gradient Boosting Classifier

Abstract

The number of people turning to the Internet to search for a diverse range of health-related subjects continues to grow and with this multitude of information available, duplicate questions become more frequent and finding the most appropriate answers becomes problematic. This issue is important for questionanswering platforms as it complicates the retrieval of all information relevant to the same topic, particularly when questions similar in essence are expressed differently, and answering a given medical question by retrieving similar questions that are already answered by human experts seems to be a promising solution. In this paper we present our novel approach to detect question entailment by determining the type of question asked rather than focusing on the type of the ailment given. This unique methodology makes the approach robust towards examples which have different ailment names but are synonyms of each other. Also it enables us to check entailment at a much more fine-grained level.

Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering

“…More generally, approaches combining ensemble methods and transfer learning of multi-task language models were the clear winners of the competition for RQE with the first and second scores (Zhu et al, 2019;Bhaskar et al, 2019). Approaches that used ensemble methods without multi-task language models (Sharma and Roychowdhury, 2019) or multi-task learning without ensemble methods (Pugaliya et al, 2019) performed worse than the first category but made it to the top 4. Domain knowledge was also used in several participating approaches with a clear positive impact.…”

“…Data augmentation also played a key role for several systems that used external data to extend batches of in-domain data (Xu et al, 2019), created synthetic data (Bannihatti Kumar et al, 2019), or used models trained on external datasets (e.g. MultiNLI) in ensemble methods (Bhaskar et al, 2019;Sharma and Roychowdhury, 2019).…”

“…Task(s) ANU-CSIRO (Nguyen et al, 2019) NLI, RQE, QA ARS NITK (Agrawal et al, 2019) NLI, RQE, QA DoubleTransfer (Xu et al, 2019) NLI, RQE, QA Dr.Quad (Bannihatti Kumar et al, 2019) NLI, RQE, QA DUT-BIM (Zhou et al, 2019a) QA DUT-NLP (Zhou et al, 2019b) RQE, QA IITP (Bandyopadhyay et al, 2019) NLI, RQE, QA IIT-KGP (Sharma and Roychowdhury, 2019) RQE KU ai (Cengiz et al, 2019) NLI lasigeBioTM (Lamurias and Couto, 2019) NLI, RQE, QA MSIT SRIB (Chopra et al, 2019) NLI NCUEE (Lee et al, 2019b) NLI PANLP (Zhu et al, 2019) NLI, RQE, QA Pentagon (Pugaliya et al, 2019) NLI, RQE, QA Saama Research (Kanakarajan, 2019) NLI Sieg (Bhaskar et al, 2019) NLI, RQE Surf (Nam et al, 2019) NLI UU TAILS (Tawfik and Spruit, 2019) NLI, RQE UW-BHI (Kearns et al, 2019) NLI WTMED (Wu et al, 2019) NLI which builds up on BERT to perform multi-task learning and is evaluated on the GLUE benchmark (Wang et al, 2018). A common theme across all the papers was training of multiple models and then using an ensemble as the final system which performed better than the individual models.…”

“…• Word Embeddings While many teams used BERT (Lamurias and Couto, 2019;Zhou et al, 2019a;Bandyopadhyay et al, 2019;Nguyen et al, 2019;Sharma and Roychowdhury, 2019) 14 , some teams also used word embeddings as the input to their models. Bhaskar et al (2019) used biomedical word embeddings from Chen et al (2018) while Kearns et al (2019) used cui2vec (Beam et al, 2018.…”

Medical Knowledge-enriched Textual Entailment Framework

“…Related Work : The development of annotated TE and NLI medical datasets (Abacha et al, 2015;Ben Abacha et al, 2019;Abacha and Demner-Fushman, 2016;Romanov and Shivade, 2018) and a variety of pre-trained language models has led to a rise of extensive ongoing research in this field. Majority of the systems developed for the TE task adopts the multi-task learning (MTL) framework (Zhu et al, 2019;Bhaskar et al, 2019;Kumar et al, 2019;Xu et al, 2019), ensemble method (Sharma and Roychowdhury, 2019), and transfer learning (Bhaskar et al, 2019) for achieving better accuracy. Xu et al (2019) employed the MTL approach Yadav et al, 2018;Yadav et al, 2019;Yadav et al, 2020) in TE task to learn from the auxiliary tasks of question answering (QA) and NLI.…”