SemEval-2017 Task 3: Community Question Answering

Abstract

We describe SemEval2017 Task 3 on Community Question Answering.This year, we reran the four subtasks from SemEval-2016: (A) Question-Comment Similarity, (B) Question-Question Similarity, (C) QuestionExternal Comment Similarity, and (D) Rerank the correct answers for a new question in Arabic, providing all the data from 2015 and 2016 for training, and fresh data for testing. Additionally, we added a new subtask E in order to enable experimentation with Multi-domain Question Duplicate Detection in a larger-scale scenario, using StackExchange subforums. A total of 23 teams participated in the task, and submitted a total of 85 runs (36 primary and 49 contrastive) for subtasks A-D. Unfortunately, no teams participated in subtask E. A variety of approaches and features were used by the participating systems to address the different subtasks. The best systems achieved an official score (MAP) of 88. 43, 47.22, 15.46, and 61.16 in subtasks A, B, C, and D, respectively. These scores are better than the baselines, especially for subtasks A-C.

Talla at SemEval-2017 Task 3: Identifying Similar Questions Through Paraphrase Detection

“…All code was implemented in Python 3.5. For data extraction, we converted the XML documents provided by (Nakov et al, 2017) into pandas DataFrames, retaining the subject text, body text, and metadata related to the original and related questions. The feature extraction and the pairwise-preference learning phase are described below.…”

UINSUSKA-TiTech at SemEval-2017 Task 3: Exploiting Word Importance Levels for Similarity Features for CQA

“…For tuning the parameters and seeking the best combination of features, we train SVM with a linear kernel on TRAIN dataset, and applied the model on DEV dataset. We choose two best cost-parameters C with specific feature combinations in Table 2 (Nakov et al, , 2017.…”

LearningToQuestion at SemEval 2017 Task 3: Ranking Similar Questions by Learning to Rank Using Rich Features

“…One way to solve this problem is to design systems to automatically find similar content (question, answer, comment) to the user's posted question. SemEval-2017 task 3 (Nakov et al, 2017) focuses on solving this problem in community question answer by various subtasks of ranking relevant information in Qatar living forums data. The system presented in this paper focuses on subtask B, to re-rank given set of questions retrieved by search engine, in their similarity to original question.…”