Using graded-relevance metrics for evaluating community QA answer selection

Sakai, Tetsuya; Ishikawa, Daisuke; Kando, Noriko; Seki, Yohei; Kuriyama, Kazuko; Lin, Chin-Yew

doi:10.1145/1935826.1935864

Cited by 40 publications

(28 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar observations have been made in evaluating CQA answer ranking 24) . While we cannot conclude how many assessors are required for reliable experiments from our present study, it is possible that ensuring the quality of annotators is more important than the number of annotators 24) .…”

Section: Classification Results With Different Annotation Datasupporting

confidence: 77%

Community QA Question Classification: Is the Asker Looking for Subjective Answers or Not?

Aikawa

Sakai

Yamana

2011

IPSJ Online Transactions

Self Cite

View full text Add to dashboard Cite

Community question answering (CQA) sites such as Yahoo! Chiebukuro are known to be very useful resources for automatic question answering (QA) systems. However, CQA users often post questions expecting not some general truths but rather opinions of different people. We believe that a QA system should act according to these different question types. We therefore define two question types based on whether the questioner expects subjective or objective answers, and report on an automatic question classification experiment. We achieve over 80% weighted accuracy using uni-gram and bi-gram features learned by Naïve Bayes with smoothing. We also discuss the inter-annotator agreement and its impact on automatic classification accuracy, as well as what kind of questions tend to be misclassified.

show abstract

Section: Classification Results With Different Annotation Datasupporting

confidence: 77%

Community QA Question Classification: Is the Asker Looking for Subjective Answers or Not?

Aikawa

Sakai

Yamana

2011

IPSJ Online Transactions

Self Cite

View full text Add to dashboard Cite

show abstract

“…Request permissions from permissions@acm.org. SIGIR'14, July [6][7][8][9][10][11]2014 level measures one would need to manually assess all possible passages, which is not practical. As an alternative, character-based measures have been developed that treat each character as a document and evaluate them using existing precision and recall measures [1,5].…”

Section: Introductionmentioning

confidence: 99%

Evaluating answer passages using summarization measures

Keikha

Park

Croft

2014

Proceedings of the 37th International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

View full text Add to dashboard Cite

Passage-based retrieval models have been studied for some time and have been shown to have some benefits for document ranking. Finding passages that are not only topically relevant, but are also answers to the users' questions would have a significant impact in applications such as mobile search. To develop models for answer passage retrieval, we need to have appropriate test collections and evaluation measures. Making annotations at the passage level is, however, expensive and can have poor coverage. In this paper, we describe the advantages of document summarization measures for evaluating answer passage retrieval and show that these measures have high correlation with existing measures and human judgments.

show abstract

“…Fichman found that the quality of answers was significantly improved only in terms of answer completeness and verifiability, rather than the answer accuracy. Sakai et al [13] proposed system evaluation methods for the task of selecting or ranking answers for a given question. They noticed that the asker-selected best answers might be biased and even if they were not, there might be other good answers besides the best ones.…”

Section: Retrieving High-quality Answers In Cqa Sitesmentioning

confidence: 99%

The Best Answers? Think Twice: Identifying Commercial Campagins in the CQA Forums

Chen

Srinivasan

et al. 2015

J. Comput. Sci. Technol.

View full text Add to dashboard Cite

In an emerging trend, more and more Internet users search for information from Community Question and Answer (CQA) websites, as interactive communication in such websites provides users with a rare feeling of trust. More often than not, end users look for instant help when they browse the CQA websites for the best answers. Hence, it is imperative that they should be warned of any potential commercial campaigns hidden behind the answers. Existing research focuses more on the quality of answers and does not meet the above need. Textual similarities between questions and answers are widely used in previous research. However, this feature will no longer be effective when facing commercial paid posters. More context information, such as writing templates and a user's reputation track, needs to be combined together to form a new model to detect the potential campaign answers. In this paper, we develop a system that automatically analyzes the hidden patterns of commercial spam and raises alarms instantaneously to end users whenever a potential commercial campaign is detected. Our detection method integrates semantic analysis and posters' track records and utilizes the special features of CQA websites largely different from those in other types of forums such as microblogs or news reports. Our system is adaptive and accommodates new evidence uncovered by the detection algorithms over time. Validated with real-world trace data from a popular Chinese CQA website over a period of three months, our system shows great potential towards adaptive detection of CQA spams.

show abstract

Using graded-relevance metrics for evaluating community QA answer selection

Cited by 40 publications

References 13 publications

Community QA Question Classification: Is the Asker Looking for Subjective Answers or Not?

Community QA Question Classification: Is the Asker Looking for Subjective Answers or Not?

Evaluating answer passages using summarization measures

The Best Answers? Think Twice: Identifying Commercial Campagins in the CQA Forums

Contact Info

Product

Resources

About