A nonparametric term weighting method for information retrieval based on measuring the divergence from independence

Kocabaş, İ̇lker; Dinçer, Bekir Taner; Karaoğlan, Bahar

doi:10.1007/s10791-013-9225-4

Cited by 24 publications

(11 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this subsection, we compare the empirical threshold of 70% with 10% by performing an extrinsic evaluation on the ClueWeb09 dataset using four different term-weighting models, namely BM25 [21], DFI [22], LGD [23] and DPH [24]. We also include a risk-sensitive evaluation similar to that of Gallagher et al [6], which takes into account not only the average effectiveness but also the per-query differences.…”

Section: Discussionmentioning

confidence: 99%

An intrinsic evaluation of the Waterloo spam rankings of the ClueWeb09 and ClueWeb12 datasets

Yılmazel

Arslan

2019

Journal of Information Science

View full text Add to dashboard Cite

The ClueWeb09 dataset and its successor, the ClueWeb12 dataset, are two of the largest collections of Web pages released by Text REtrieval Conference (TREC). The ClueWeb datasets were used in various tracks of TREC ran through 2009 to 2017. For every year, approximately 50 new queries are released and a pool of Web pages are judged against these queries by human assessors as relevant, non-relevant or spam. In this article, a ground truth for binary classification (spam vs non-spam) is constructed from Web pages that are judged as spam or relevant under the assumption that a Web page judged as relevant for any query cannot be spam. Based on this ground truth, we evaluate classification performances of the Waterloo spam rankings (Fusion, Britney, GroupX and UK2006), which have been traditionally used to identify and filter spam pages in retrieval systems. The experimental results in terms of the universal binary classification evaluation measures suggest that the Fusion (with threshold = 11%) is the best for the ClueWeb09 dataset. Analysis of the frequency distributions of relevant/spam documents over spam scores reveals that the GroupX is the most powerful at identifying relevant documents, whereas the Fusion is the most powerful at identifying spam documents. It is also confirmed that the effectiveness of the Fusion spam ranking of the ClueWeb12 dataset is not as good as that of the ClueWeb09.

show abstract

Section: Discussionmentioning

confidence: 99%

An intrinsic evaluation of the Waterloo spam rankings of the ClueWeb09 and ClueWeb12 datasets

Yılmazel

Arslan

2019

Journal of Information Science

View full text Add to dashboard Cite

show abstract

“…The most correlated feature (AP and NDCG) is WMODEL.DFIZ_std which represents a weighting model based on the standardized distance from independence in term frequency [16]. WMODEL.ML2_std (second feature for AP) represents a weighting model based on multinomial randomness model, with Laplace after-e ect model and normalisation 2 [12]. WMODEL.In_expC2_max (second feature for NDCG) is the Inverse Expected Document Frequency weighting model with Bernoulli after-e ect and normalization [1].…”

Section: Individual E Ectivenessmentioning

confidence: 99%

Query Performance Prediction Focused on Summarized Letor Features

Chifu

Laporte

Mothe

et al. 2018

The 41st International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

View full text Add to dashboard Cite

Query performance prediction (QPP) aims at automatically estimating the information retrieval system effectiveness for any user's query. Previous work has investigated several types of pre-and post-retrieval query performance predictors; the latter has been shown to be more effective. In this paper we investigate the use of features that were initially defined for learning to rank in the task of QPP. While these features have been shown to be useful for learning to rank documents, they have never been studied as query performance predictors. We developed more than 350 variants of them based on summary functions. Conducting experiments on four TREC standard collections, we found that Letor-based features appear to be better QPP than predictors from the literature. Moreover, we show that combining the best Letor features outperforms the state of the art query performance predictors. This is the first study that considers such an amount and variety of Letor features for QPP and that demonstrates they are appropriate for this task.

show abstract

“…vestigated the use of the Student's t-test for risk-sensitive evaluation, this is the first work to investigate the use of Pearson's Chi-square statistic for risk-sensitive evaluation, thereby facilitating the use of multiple baselines. Instead, previous usages of the Chi-square statistic has encompassed index term weighing [18] and document classification [22].…”

Section: Related Workmentioning

confidence: 99%

“…SIGIR '16, July [17][18][19][20][21]2016, Pisa, Italy. sure should consider per-query losses and gains compared to a particular baseline technique [11].…”

Section: Introductionmentioning

confidence: 99%

Risk-Sensitive Evaluation and Learning to Rank using Multiple Baselines

Dinçer

Macdonald

Ounis

2016

Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

A robust retrieval system ensures that user experience is not damaged by the presence of poorly-performing queries. Such robustness can be measured by risk-sensitive evaluation measures, which assess the extent to which a system performs worse than a given baseline system. However, using a particular, single system as the baseline suffers from the fact that retrieval performance highly varies among IR systems across topics. Thus, a single system would in general fail in providing enough information about the real baseline performance for every topic under consideration, and hence it would in general fail in measuring the real risk associated with any given system. Based upon the Chi-squared statistic, we propose a new measure Z Risk that exhibits more promise since it takes into account multiple baselines when measuring risk, and a derivative measure called GeoRisk, which enhances Z Risk by also taking into account the overall magnitude of effectiveness. This paper demonstrates the benefits of Z Risk and GeoRisk upon TREC data, and how to exploit GeoRisk for risk-sensitive learning to rank, thereby making use of multiple baselines within the learning objective function to obtain effective yet risk-averse/robust ranking systems. Experiments using 10,000 topics from the MSLR learning to rank dataset demonstrate the efficacy of the proposed Chi-square statistic-based objective function.

show abstract

A nonparametric term weighting method for information retrieval based on measuring the divergence from independence

Cited by 24 publications

References 29 publications

An intrinsic evaluation of the Waterloo spam rankings of the ClueWeb09 and ClueWeb12 datasets

An intrinsic evaluation of the Waterloo spam rankings of the ClueWeb09 and ClueWeb12 datasets

Query Performance Prediction Focused on Summarized Letor Features

Risk-Sensitive Evaluation and Learning to Rank using Multiple Baselines

Contact Info

Product

Resources

About