Intra-assessor consistency in question answering

Ruthven, Ian; Glasgow, L; Baillie, Mark; Bierig, Ralf; Nicol, Emma; Sweeney, Simon; Yakici, Murat

doi:10.1145/1277741.1277879

Cited by 5 publications

(8 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the idea of relevance being inherently subjective has been pointed out by many researchers (e.g., see references [29] and more recently [21]), we note that in community QA a large fraction of the questions are subjective, compounding the problem of both relevance assessment (which is no longer meaningful). Information seeker satisfaction has been studied in ad-hoc IR context in [11] (refer to [15] for an overview), but studies have been limited by lack of realistic user feedback on whole-result satisfaction and instead worked primarily within the Cranfield evaluation model.…”

Section: Related Workmentioning

confidence: 78%

“…This is in contrast to the more traditional relevance-based assessment that is often done by judges different from the original information seeker, which may result in ratings that do not agree with the target user. While the idea of relevance being inherently subjective has been pointed out in the past (e.g., see references [29] and more recently [21]), nowhere does the problem of subjective relevance arise more prominently than within Community QA, where many of the questions are inherently subjective, complex, ill-formed, or often all of the above. The problem of complex and subjective QA has only recently started to be addressed in the question answering community, most recently as the first opinion QA track in TREC [7].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Predicting information seeker satisfaction in community question answering

Liu

Bian

Agichtein

2008

Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

200

141

View full text Add to dashboard Cite

Question answering communities such as Naver and Yahoo! Answers have emerged as popular, and often effective, means of information seeking on the web. By posting questions for other participants to answer, information seekers can obtain specific answers to their questions. Users of popular portals such as Yahoo! Answers already have submitted millions of questions and received hundreds of millions of answers from other participants. However, it may also take hours -and sometime days-until a satisfactory answer is posted. In this paper we introduce the problem of predicting information seeker satisfaction in collaborative question answering communities, where we attempt to predict whether a question author will be satisfied with the answers submitted by the community participants. We present a general prediction model, and develop a variety of content, structure, and community-focused features for this task. Our experimental results, obtained from a largescale evaluation over thousands of real questions and user ratings, demonstrate the feasibility of modeling and predicting asker satisfaction. We complement our results with a thorough investigation of the interactions and information seeking patterns in question answering communities that correlate with information seeker satisfaction. Our models and predictions could be useful for a variety of applications such as user intent inference, answer ranking, interface design, and query suggestion and routing.

show abstract

Section: Related Workmentioning

confidence: 78%

Section: Introductionmentioning

confidence: 99%

Predicting information seeker satisfaction in community question answering

Liu

Bian

Agichtein

2008

Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

200

141

View full text Add to dashboard Cite

show abstract

“…While the idea of relevance being inherently subjective has been pointed out by many researchers (e.g., see Zobel [1998] and more recently Ruthven et al [2007]), we note that in community QA a large fraction of the questions are subjective, compounding the problem of both relevance assessment (which is no longer meaningful). Information seeker satisfaction has been studied in ad-hoc IR context in Harter and Hert [1997] (refer to Kobayashi and Takeda [2000] for an overview), but studies have been limited by lack of realistic user feedback on whole-result satisfaction and instead worked primarily within the Cranfield evaluation model.…”

Section: Related Workmentioning

confidence: 78%

“…This is in contrast to the more traditional relevance-based assessment that is often done by judges different from the original information seeker, which may result in ratings that do not agree with the target user. While the idea of relevance being inherently subjective has been pointed out in the past (e.g., see Zobel [1998] and more recently Ruthven et al [2007]), nowhere does the problem of subjective relevance arise more prominently than within Community QA, where many of the questions are inherently subjective, complex, ill-formed, or often all of the above. The problem of complex and subjective QA has only recently started to be addressed in the question answering community, most recently as the first opinion QA track in TREC [Dang et al 2007].…”

Section: Introductionmentioning

confidence: 99%

Modeling information-seeker satisfaction in community question answering

Agichtein

Liu

Bian

2009

ACM Trans. Knowl. Discov. Data

View full text Add to dashboard Cite

Question Answering Communities such as Naver, Baidu Knows, and Yahoo! Answers have emerged as popular, and often effective, means of information seeking on the web. By posting questions for other participants to answer, information seekers can obtain specific answers to their questions. Users of CQA portals have already contributed millions of questions, and received hundreds of millions of answers from other participants. However, CQA is not always effective: in some cases, a user may obtain a perfect answer within minutes, and in others it may require hours-and sometimes days-until a satisfactory answer is contributed. We investigate the problem of predicting information seeker satisfaction in collaborative question answering communities, where we attempt to predict whether a question author will be satisfied with the answers submitted by the community participants. We present a general prediction model, and develop a variety of content, structure, and community-focused features for this task. Our experimental results, obtained from a largescale evaluation over thousands of real questions and user ratings, demonstrate the feasibility of modeling and predicting asker satisfaction. We complement our results with a thorough investigation of the interactions and information seeking patterns in question answering communities that correlate with information seeker satisfaction. We also explore personalized models of asker satisfaction, and show that when sufficient interaction history exists, personalization can significantly improve prediction accuracy over a "one-size-fits-all" model. Our models and predictions could be useful for a variety of applications, such as user intent inference, answer ranking, interface design, and query suggestion and routing.

show abstract

“…Second, the quality of answer lists shown in Figure 1 includes the primary assessor's judgment after the interaction. Ideally, the interactive assessor's judgment at the individual nugget level should be obtained at the same time that they interact with the side-by-side interface, since judgments may change as a topic becomes more and more familiar [3]. This is also borne out by a related experiment: as part of our participation in the ciQA task, we also used another interface to gather assessor judgments of individual answer strings.…”

Section: Discussionmentioning

confidence: 99%

User preference choices for complex question answering

Scholer

Turpin

2008

Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Question answering systems increasingly need to deal with complex information needs that require more than simple factoid answers. The evaluation of such systems is usually carried out using precision-or recall-based system performance metrics. Previous work has demonstrated that when users are shown two search result lists side-by-side, they can reliably differentiate between the qualities of the lists. We investigate the consistency between this user-based approach and system-oriented metrics in the question answering environment. Our initial results indicate that the two methodologies show a high level of disagreement.

show abstract

Intra-assessor consistency in question answering

Abstract: In this paper we investigate the consistency of answer assessment in a complex question answering task examining features of assessor consistency, types of answers and question type.

Cited by 5 publications

References 3 publications

Predicting information seeker satisfaction in community question answering

Predicting information seeker satisfaction in community question answering

Modeling information-seeker satisfaction in community question answering

User preference choices for complex question answering

Contact Info

Product

Resources

About