Including summaries in system evaluation

Turpin, Andrew; Scholer, Falk; Järvelin, Kalvero; Wu, Mingfang; Culpepper, J. Shane

doi:10.1145/1571941.1572029

Cited by 39 publications

(36 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Highly relevant documents are effective in RFB [3] and users can readily recognize them in search results [12]. Apparently relevant query-biased summaries are also good indicators of document relevance [11]. Thus users could provide effective feedback and summaries would be effective sources of search keys.…”

Section: Introductionmentioning

confidence: 99%

Interactive relevance feedback with graded relevance and sentence extraction

Järvelin

2009

Proceedings of the 18th ACM Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Research on relevance feedback (RFB) in information retrieval (IR) has given mixed results. Success in RFB seems to depend on the searcher's willingness to provide feedback and ability to identify relevant documents or query keys. The paper is based on simulating many user scenarios regarding the amount and quality of RFB. In addition, we experiment with query-biased sentence extraction for query reformulation. The baselines are initial nofeedback queries and queries based on pseudo-relevance feedback. The core question is: under which conditions would RFB based on sentence extraction be successful? The answer depends on user's behavior, implementation of feedback query formulation, and the evaluation methods. A small amount of feedback from a short browsing window seems to improve the final ranking the most. Longer browsing allows more feedback and better queries but also consumes the available relevant documents.

show abstract

Section: Introductionmentioning

confidence: 99%

Interactive relevance feedback with graded relevance and sentence extraction

Järvelin

2009

Proceedings of the 18th ACM Conference on Information and Knowledge Management

View full text Add to dashboard Cite

show abstract

“…The reason for using unnormalized version of the metrics is that the total number of relevant documents is unknown. The second group of metrics have the same forms as those in the first group, but one document is considered to be relevant iff the document and its snippet are both relevant [13]. The third groups of metrics are effective time ratio and its extensions.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…On the contrary, users may miss relevant documents or waste time clicking and examining irrelevant documents due to bad snippets. Turpin et al [13] investigated this problem and showed that it makes difference by including snippet quality in search engine evaluation. In this paper, we interpret traditional IR metric precision as effective time ratio of the real user, i.e., the ratio between the time used in reading relevant information and the total search time, and extend it in the scenario when the search engines provide document snippets.…”

Section: Beyond Document Relevancementioning

confidence: 99%

Evaluating Search Engines by Clickthrough Data

2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. It is no doubt that search is critical to the web. And it will be of similar importance to the semantic web. Once searching from billions of objects, it will be impossible to always give a single right result, no matter how intelligent the search engine is. Instead, a set of possible results will be provided for the user to choose from. Moreover, if we consider the trade-off between the system costs of generating a single right result and a set of possible results, we may choose the latter. This will naturally lead to the question of how to decide on and present the set to the user and how to evaluate the outcome.In this paper, we introduce some new methodology in evaluation of web search technologies and systems. Historically, the dominant method for evaluating search engines is the Cranfield paradigm, which employs a test collection to qualify the systems' performance. However, the modern search engines are much different from the IR systems when the Cranfield paradigm was proposed: 1) Most modern search engines have much more features, such as snippets and query suggestions, and the quality of such features can affect the users' utility; 2) The document collections used in search engines are much larger than ever, so the complete test collection that contains all query-document judgments is not available. As response to the above differences and difficulties, the evaluation based on implicit feedback is a promising alternative employed in IR evaluation. With this approach, no extra human effort is required to judge the query-document relevance. Instead, such judgment information can be automatically predicted from real users' implicit feedback data. There are three key issues in this methodology: 1) How to estimate the query-document relevance and other useful features that useful to qualify the search engine performance; 2) If the complete "judgments" are not available, how can we efficiently collect the most critical information from which the system performance can be derived; 3) Because query-document relevance is not only feature that can affect the performance, how can we integrate others to be a good metric to predict the system performance. We will show a set of technologies dealing with these issues.

show abstract

“…Our model of user behavior is an extension of the existing behavior modeled by other metrics that incorporate document summaries [5,12,14] with the important addition of time. Viewing a summary and deciding to click on it takes a certain amount of time.…”

Section: Stochastic Simulationmentioning

confidence: 99%

Stochastic simulation of time-biased gain

Smucker

Clarke

2012

Proceedings of the 21st ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Time-biased gain provides a unifying framework for information retrieval evaluation, generalizing many traditional effectiveness measures while accommodating aspects of user behavior not captured by these measures. By using time as a basis for calibration against actual user data, timebiased gain can reflect aspects of the search process that directly impact user experience, including document length, near-duplicate documents, and summaries. Unlike traditional measures, which must be arbitrarily normalized for averaging purposes, time-biased gain is reported in meaningful units, such as the total number of relevant documents seen by the user. In prior work, we proposed and validated a closed-form equation for estimating time-biased gain, explored its properties, and compared it to standard approaches. In this paper, we use stochastic simulation to numerically approximate time-biased gain. Stochastic simulation provides greater flexibility that will allow us, in future work, to easily accommodate different types of user behavior and increase the realism of the effectiveness measure.

show abstract

Including summaries in system evaluation

Cited by 39 publications

References 21 publications

Interactive relevance feedback with graded relevance and sentence extraction

Interactive relevance feedback with graded relevance and sentence extraction

Evaluating Search Engines by Clickthrough Data

Stochastic simulation of time-biased gain

Contact Info

Product

Resources

About