PMI-cool at SemEval-2016 Task 3: Experiments with PMI and Goodness Polarity Lexicons for Community Question Answering

Balchev, Daniel; Kiprov, Yasen; Koychev, Ivan; Nakov, Preslav

doi:10.18653/v1/s16-1130

Cited by 17 publications

(11 citation statements)

References 18 publications

(14 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, given the recent success of deep neural networks, which largely eliminate the need for manual feature engineering, we are interested in exploring various deep neural network architectures, e.g., based on LSTMs, GRUs, CNNs, RNMs, etc., as well as various ways to generate high-quality word embeddings (Mihaylov and Nakov 2016b) and goodness polarity lexicons (Balchev et al 2016; Mihaylov et al 2017), e.g., as was done for SemEval-2016 Task 3 for English. We would also like to compare side-by-side deep neural networks to convolutional syntactic kernels, and to investigate ways to combine them.…”

Section: Resultsmentioning

confidence: 99%

Arabic community question answering

Nakov¹,

Màrquez²,

Moschitti³

et al. 2018

Nat. Lang. Eng.

Self Cite

View full text Add to dashboard Cite

We analyze resources and models for Arabic community Question Answering (cQA). In particular, we focus on CQA-MD, our cQA corpus for Arabic in the domain of medical forums. We describe the corpus and the main challenges it poses due to its mix of informal and formal language, and of different Arabic dialects, as well as due to its medical nature. We further present a shared task on cQA at SemEval, the International Workshop on Semantic Evaluation, based on this corpus. We discuss the features and the machine learning approaches used by the teams who participated in the task, with focus on the models that exploit syntactic information using convolutional tree kernels and neural word embeddings. We further analyze and extend the outcome of the SemEval challenge by training a meta-classifier combining the output of several systems. This allows us to compare different features and different learning algorithms in an indirect way. Finally, we analyze the most frequent errors common to all approaches, categorizing them into prototypical cases, and zooming into the way syntactic information in tree kernel approaches can help solve some of the most difficult cases. We believe that our analysis and the lessons learned from the process of corpus creation as well as from the shared task analysis will be helpful for future research on Arabic cQA.

show abstract

Section: Resultsmentioning

confidence: 99%

Arabic community question answering

Nakov¹,

Màrquez²,

Moschitti³

et al. 2018

Nat. Lang. Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Finally, we should mention some interesting features used by the participating systems across all three subtasks. This includes fine-tuned word embeddings 5 (Mihaylov and Nakov, 2016b); features modeling text complexity, veracity, and user trollness 6 (Mihaylova et al, 2016); sentiment polarity features (Nicosia et al, 2015); and PMI-based goodness polarity lexicons (Balchev et al, 2016;Mihaylov et al, 2017a).…”

Section: Related Workmentioning

confidence: 99%

SemEval-2017 Task 3: Community Question Answering

Nakov¹,

Hoogeveen²,

Màrquez³

et al. 2017

Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)

Self Cite

181

157

View full text Add to dashboard Cite

We describe SemEval2017 Task 3 on Community Question Answering.This year, we reran the four subtasks from SemEval-2016: (A) Question-Comment Similarity, (B) Question-Question Similarity, (C) QuestionExternal Comment Similarity, and (D) Rerank the correct answers for a new question in Arabic, providing all the data from 2015 and 2016 for training, and fresh data for testing. Additionally, we added a new subtask E in order to enable experimentation with Multi-domain Question Duplicate Detection in a larger-scale scenario, using StackExchange subforums. A total of 23 teams participated in the task, and submitted a total of 85 runs (36 primary and 49 contrastive) for subtasks A-D. Unfortunately, no teams participated in subtask E. A variety of approaches and features were used by the participating systems to address the different subtasks. The best systems achieved an official score (MAP) of 88. 43, 47.22, 15.46, and 61.16 in subtasks A, B, C, and D, respectively. These scores are better than the baselines, especially for subtasks A-C.

show abstract

“…Following (Balchev et al, 2016), we build this lexicon using pointwise mutual information, starting with the training data from SemEval-2016 task 3, and then extending this to words from the Qatar Living dump. We use the same nine features as for sentiment, but this time we only have one lexicon and we only use words (no bigrams).…”

Section: Answer Features Credibility (31 Features)mentioning

confidence: 99%

“…GOODNESS (9 features) Similarly, we build goodness polarity lexicons that contain 41,633 words, each associated with a real number representing its strength of association with Good or Bad answers. Following (Balchev et al, 2016), we build this lexicon using pointwise mutual information, starting with the training data from SemEval-2016 task 3, and then extending this to words from the Qatar Living dump. We use the same nine features as for sentiment, but this time we only have one lexicon and we only use words (no bigrams).…”

Section: Credibility (31 Features)mentioning

confidence: 99%

Do Not Trust the Trolls: Predicting Credibility in Community Question Answering Forums

Nakov¹,

Mihaylova²,

Màrquez³

et al. 2017

RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning

Self Cite

View full text Add to dashboard Cite

We address information credibility in community forums, in a setting in which the credibility of an answer posted in a question thread by a particular user has to be predicted. First, we motivate the problem and we create a publicly available annotated English corpus by crowdsourcing. Second, we propose a large set of features to predict the credibility of the answers. The features model the user, the answer, the question, the thread as a whole, and the interaction between them. Our experiments with ranking SVMs show that the credibility labels can be predicted with high performance according to several standard IR ranking metrics, thus supporting the potential usage of this layer of credibility information in practical applications. The features modeling the profile of the user (in particular trollness) turn out to be most important, but embedding features modeling the answer and the similarity between the question and the answer are also very relevant. Overall, half of the gap between the baseline performance and the perfect classifier can be covered using the proposed features.

show abstract

PMI-cool at SemEval-2016 Task 3: Experiments with PMI and Goodness Polarity Lexicons for Community Question Answering

Cited by 17 publications

References 18 publications

Arabic community question answering

Arabic community question answering

SemEval-2017 Task 3: Community Question Answering

Do Not Trust the Trolls: Predicting Credibility in Community Question Answering Forums

Contact Info

Product

Resources

About