Fermi at SemEval-2019 Task 8: An elementary but effective approach to Question Discernment in Community QA Forums

Abstract

Online Community Question Answering Forums (cQA) have gained massive popularity within recent years. The rise in users for such forums have led to the increase in the need for automated evaluation for question comprehension and fact evaluation of the answers provided by various participants in the forum. Our team, Fermi, participated in sub-task A of Task 8 at SemEval 2019-which tackles the first problem in the pipeline of factual evaluation in cQA forums, i.e., deciding whether a posed question asks for a factual information, an opinion/advice or is just socializing. This information is highly useful in segregating factual questions from non-factual ones which highly helps in organizing the questions into useful categories and trims down the problem space for the next task in the pipeline for fact evaluation among the available answers. Our system uses the embeddings obtained from Universal Sentence Encoder combined with XGBoost for the classification subtask A. We also evaluate other combinations of embeddings and off-the-shelf machine learning algorithms to demonstrate the efficacy of the various representations and their combinations. Our results across the evaluation test set gave an accuracy of 84% and received the first position in the final standings judged by the organizers.

AnswerFact: Fact Checking in Product Question Answering

“…From the annotation, we found that factual questions are actually the dominant type in PQA forums where around 71% of the annotated questions are factual ones. Following the strategy in Syed et al (2019) which ranked first for predicting the question type in SemEval-2019 Task 8, we applied the Universal Sentence representation (Cer et al, 2018) to encode question texts. While we found that the SVM classifier performs slightly better than the XGBoost (Chen and Guestrin, 2016) used in their work, achieving average 0.85 accuracy and 0.90 F1 score under the 5-fold cross validation.…”

SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums

“…Fermi (Syed et al, 2019) IIIT Hyderabad, Microsoft, Teradata 0.840 0.7182 0.7353 TMLab (Niewiński et al, 2019) Samsung R&D Institute, Warsaw, Poland The best system for Subtask A was by team Fermi (IIIT Hyderabad). They used Google's Universal Sentence representation (Cer et al, 2018), and XGBoost (Chen and Guestrin, 2016).…”

“…Fermi (Syed et al, 2019) IIIT Hyderabad, Microsoft, Teradata 0.840 0.7182 0.7353 TMLab (Niewiński et al, 2019) Samsung (Some teams did not submit system description papers, and thus we have no citations for their systems.) ablation studies, and the experiments with different techniques are described by the participants in their respective system description papers.…”